Generating augmented reality experiences utilizing physical objects to represent analogous virtual objects

ABSTRACT

This disclosure describes embodiments of methods, non-transitory computer-readable media, and systems for detecting that a physical space includes a physical object corresponding to an analogous virtual object from an augmented reality experience and rendering (or otherwise modifying) the augment reality experience to integrate the physical object as part of the experience. In particular, the disclosed systems can determine that a physical object within a physical environment corresponds to an analogous virtual object of an augmented reality experience. Based on this correspondence, the disclosed systems can modify one or more of the virtual graphics, sound, or other features corresponding to the augmented reality experience to represent the virtual object using the physical object. For example, the disclosed systems can modify acoustic features of a sound for the augmented reality experience to simulate the sound originating from (or being affected by) the physical object.

BACKGROUND

In recent years, augmented reality systems have significantly improvedthe realism and detail of virtual imagery. For example, existingaugmented reality systems can generate colorful and interactiveaugmented reality experiences that overlay virtual objects over realphysical environments. In some cases, an existing augmented realitysystem can generate an interactive augmented reality experience for agame or simulation, where the experience includes virtual objectspositioned at specific locations within a physical space. A user of anaugmented-reality-computing device can view and interact with suchvirtual objects as part of the game or simulation.

Although conventional augmented reality systems can generate engagingand realistic augmented reality experiences, such systems often consumeexcessive computer processing, memory, or other computing resources toproduce the realism and detail of today's augmented reality experiences.To generate a single frame of a virtual experience, for instance, someexisting augmented reality systems consume much of the processing powerof a Graphics Processing Unit (“GPU”) to render the frame with highresolutions of 1920 by 1080 pixels (or greater). Because a GPU oftenprocesses at speeds slower than a general Central Processing Unit(“CPU”), some existing augmented reality systems lack the processingpower to render realistic virtual objects or entire augmented-realityexperiences in real (or near-real) time.

In addition to consuming significant processing power, some existingaugmented reality systems inefficiently transfer memory between mainmemory (e.g., host memory) and GPU dedicated memory (e.g., devicememory). For example, because GPUs generally operate at a much lowerclock speed than a CPU in existing augmented reality systems, transfersbetween host memory and device memory often have limited bandwidth andhigh latency. This performance bottleneck results in poorly optimizedGPU-acceleration applications, such as when existing augmented realitysystems generate augmented reality experiences.

Such processing speeds and memory transfers become even more difficultwhen existing augmented reality systems use a head-mounted device, amobile computing device, or other smaller computing devices to renderaugmented reality experiences. Because computing devices require suchprocessing and memory to extemporaneously render augmented reality, someaugmented reality systems execute programs designed to produce lowerresolution and less realistic virtual objects.

Beyond the computing-resource demands of virtual graphics, some existingaugmented reality systems consume significant computing resources byincorporating sound into augmented reality experiences. For example,existing augmented reality systems utilize excessive processing andmemory in altering sounds to simulate those sounds coming from virtualobjects in an augmented reality experience. In comparison to complexsounds produced by physical objects (e.g., the complex sound of a carengine that includes multiple sound components), existing augmentedreality systems consume increased computing resources in attempting tosimulated complex sounds coming from a virtual object. For example, someexisting systems waste significant computing resources in generatingmultiple audio streams corresponding to the complex sound and thenaltering each audio stream to simulate origination from a virtualobject—all to complete the illusion that the virtual object is creatingthe complex sound in the same way that a similar physical object wouldcreate the same sound.

As suggested by the computing-resource demands described above, byrigidly rendering virtual object after virtual object—frame afterframe—existing augmented reality systems can consume loads of processingpower and memory for augmented reality experiences in common physicalenvironments. In some cases, augmented reality systems perform the samealgorithms and computer processing to map a physical space and renderthe same virtual objects—even when a computing device has previouslyencountered the physical space and its constituent physical objects.Despite one computing device or another computing device mapping acommon physical object or rendering common virtual objects, someconventional augmented reality systems often operate in isolation and donot save previously three-dimensional mappings or share such mappings orother calculations with other computing devices that may share the samephysical space or virtual objects.

SUMMARY

This disclosure describes one or more embodiments of methods,non-transitory computer-readable media, and systems that solve theforegoing problems or provide other benefits. For instance, thedisclosed systems can detect that a physical space includes a physicalobject corresponding to an analogous virtual object from an augmentedreality experience and present the augment reality experience byanchoring or changing a sound—or modifying graphics—for the augmentedreality experience to simulate the physical object as part of theexperience. In particular, the disclosed systems can determine that aphysical object within a physical environment corresponds to ananalogous virtual object of an augmented reality experience. Based onthis correspondence, the disclosed systems can modify one or more of thevirtual graphics, sound, or other features corresponding to theaugmented reality experience to represent the virtual object using thephysical object. To integrate the physical object into the augmentedreality experience, the disclosed systems can modify acoustic featuresof a sound for the augmented reality experience to simulate the soundoriginating from (or being affected by) the physical object.Additionally or alternatively, the disclosed systems can modify or omitvirtual graphics to depict the physical object as part of the augmentedreality experience and extemporaneously modify the augmented realityexperience based on user interactions with the physical object orcorresponding virtual graphic.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description provides one or more embodiments withadditional specificity and detail through the use of the accompanyingdrawings, as briefly described below.

FIG. 1 illustrates an example environment in which an augmented realitysystem can operate in accordance with one or more embodiments.

FIG. 2 illustrates an overview of an augmented reality systemdetermining a physical object corresponds to an analogous virtual objectfor an augmented reality experience and presenting the augmented realityexperience by modifying one or more features of the experience tointegrate the physical object in accordance with one or moreembodiments.

FIG. 3A illustrates a view of an augmented reality system mapping aphysical environment and localizing an augmented-reality-computingdevice in accordance with one or more embodiments.

FIG. 3B illustrates a schematic of an augmented reality systemleveraging the process by which a user localizes a sound in accordancewith one or more embodiments.

FIGS. 4A-4D illustrate an augmented reality system modifying one or bothof graphic and acoustic features of an augmented reality experience tointegrate a physical object within the experience in accordance with oneor more embodiments.

FIGS. 5A-5B illustrate an augmented reality system modifying acousticfeatures of an augmented reality experience based on physicalcharacteristics of a physical object in accordance with one or moreembodiments.

FIGS. 6A-6E illustrate an augmented reality system modifying graphic orinteractive features of an augmented reality experience to integrate aphysical object in accordance with one or more embodiments.

FIG. 7 illustrates a schematic diagram of an augmented reality system inaccordance with one or more embodiments.

FIG. 8 illustrates a flowchart of a series of acts for determining aphysical object from a physical environment corresponds to an analogousvirtual object for an augmented reality experience and modifyingacoustic features of a sound for the augmented reality experience tointegrate the physical object into the augmented reality experience inaccordance with one or more embodiments.

FIG. 9 illustrates a block diagram of an exemplary computing device inaccordance with one or more embodiments.

FIG. 10 is an example network environment of an augmented reality systemin accordance with one or more embodiments.

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of an augmentedreality system that detects a physical object from a physicalenvironment corresponds to an analogous virtual object from an augmentedreality experience and presents the augment reality experience byanchoring or changing a sound for the augmented reality experience—ormodifying or removing graphics representing the analogous virtual objectfor the augmented reality experience—to integrate the physical objectinto the augmented reality experience. For example, the augmentedreality system can anchor acoustic features (or other features) of asound for the augmented reality experience to a physical objectcorresponding to an analogous virtual object from the augmented realityexperience. The augmented reality system can further generate or modifygraphical features of virtual objects to simulate the physical object asan interactive part of the augmented reality experience. By anchoring orchanging a sound to integrate a physical object into an augmentedreality experience without (or instead of) an analogous virtual object,the augmented reality system efficiently renders graphics or generatessound for the augmented reality experience—thereby reducing the computerprocessing and other computing resources for conventionally renderingsuch an experience.

In some embodiments, for example, the augmented reality system capturesa data stream corresponding to a physical environment utilizing anaugmented-reality-computing device, such as a head-mounted-displaydevice, a smart phone, or a smart tablet. By analyzing the captured datastream, the augmented reality system determines that a physical objectin the physical environment corresponds to an analogous virtual objectof an augmented reality experience. The augmented reality system canthen signal or otherwise trigger the augmented-reality-computing deviceto present the augmented reality experience without utilizing theanalogous virtual object. In some cases, for instance, theaugmented-reality-computing device renders an augmented reality scenefor display utilizing the physical object instead of the analogousvirtual object. While presenting the augmented reality experience, theaugmented reality system can modify acoustic features of a sound for theaugmented reality experience to simulate either the sound originatingfrom the physical object or an effect on the sound by the physicalobject. Additionally, or alternatively, the augmented reality system canmodify or remove virtual graphics representing (or part of) theanalogous virtual object for the augmented reality experience tointegrate the physical object into the augmented reality experience.

To further illustrate, the augmented reality system can capture a datastream, corresponding to a physical environment, such as an image data,audio data, or data capture by environmental sensors. The augmentedreality system can further map the physical environment relative to theaugmented-reality-computing device to identify candidate physicalobjects within the physical environment. For example, the augmentedreality system can map the physical environment to determine spatialrelationships between features and objects of the physical environment(e.g., walls, furniture, windows, books, toys) and theaugmented-reality-computing device. The augmented reality system canfurther recognize and analyze the physical objects within the physicalenvironment to determine object types, object classifications, objectfeatures, and/or object characteristics.

In one or more embodiments, the augmented reality system furtherdetermines physical objects detected within a physical environment areanalogous to virtual objects within a corresponding augmented realityexperience. The physical object need not be identical to an analogousvirtual object but share common visual characteristics. For example, theaugmented reality system can analyze virtual objects within (or as partof) the augmented reality experience to determine types,classifications, features, and characteristics of the virtual objects.The physical object may also share functional characteristics with ananalogous virtual object. For example, the augmented reality system cananalyze virtual objects within the augmented reality experience todetermine a function of one or more virtual objects. In some cases, theaugmented reality system determines a physical object displays one ormore images or produces audio as a function corresponding to ananalogous virtual object. To illustrate, the augmented reality systemcan determine that (i) a function of a physical stereo system is toproduce music or other auditory sounds similar to a virtual stereosystem or that (ii) a function of a physical television or displayscreen is to display images similar to a virtual display screen.

In at least one embodiment, the augmented reality system can furtheridentify analogous virtual objects by determining threshold matchesbetween the types, classifications, features, functions, andcharacteristics of the physical objects and the virtual objects. Forinstance, the augmented reality system can determine a physical objectmatches an analogous virtual object based on an object-matching score orother appropriate techniques.

Upon detecting a physical object corresponds to an analogous virtualobject from an augmented reality experience, the augmented realitysystem can present the augmented reality experience without some or allof the analogous virtual object. For example, the augmented realitysystem can generate, render, or otherwise present the augmented realityexperience without utilizing the analogous virtual object. In somecases, the augmented reality system can render the augmented realityexperience utilizing the physical object instead of the analogousvirtual object. In some embodiments, the augmented reality systemrenders a portion of the analogous virtual object as an overlay on thecorresponding physical object.

In addition to presenting an augmented reality experience withoututilizing the analogous virtual object and instead utilizing a detectedphysical object, the augmented reality system can further anchoracoustic features of a sound or graphical features for the augmentedreality experience to the physical object. For example, the augmentedreality system can anchor or change acoustic features of a sound for theaugmented reality experience to the physical object to simulate eitherthe sound originating from the physical object or an effect on the soundby the physical object. To illustrate, the augmented reality system cananchor acoustic features of music for an augmented reality experience toa physical speaker identified in a physical environment. In one or moreembodiments, the augmented reality system anchors acoustic or graphicalfeatures of the augmented reality experience by associating a locationof the physical object with the anchored acoustic or graphical feature,such that any display, playback, or presentation associated with thatfeature within the augmented reality experience appears to originatefrom (or be affected by) the location and other characteristics of thephysical object.

As indicated above, in certain implementations, the augmented realitysystem modifies acoustic features of a sound for an augmented realityexperience to simulate either the sound originating from a physicalobject or an effect on the sound by the physical object. For example,the augmented reality system can modify acoustic features of the soundbased on a distance and angle between the location of the physicalobject to which the sound is anchored and theaugmented-reality-computing device. The augmented reality system canadditionally modify acoustic features of the sound based on spectrallocalization cues that inform how the user understands the location ofthe sound, as well as on visual characteristics of the anchored physicalobject that may affect how the sound is heard (e.g., the size of thephysical object, the direction that the physical object is pointing). Inat least one embodiment, the augmented reality system can simulatesounds to be affected by a physical property of the physical object,such as with a sound that is altered to simulate that the soundoriginates outside of a window within the physical environment.

To further or otherwise enhance the augmented reality experience, theaugmented reality system can modify graphical features of the augmentedreality experience corresponding to the physical object for displaywithin the physical environment. For example, the augmented realitysystem can generate a full or partial overlay for the physical objectbased on the analogous virtual object. In some cases, the augmentedreality system generates a graphical overlay appearing similar to theanalogous virtual object to modify the appearance of the physical objectto simulate the analogous virtual object. The augmented reality systemcan further position the graphical overlay at the location of thephysical object within the augmented reality experience. By positioningthe graphical overlay in this manner, the augmented reality system canpartially or completely obscure the underlying physical object, such asby giving a physical book a new virtual cover.

In one or more embodiments, the augmented reality system can also trackuser motions and interactions with (or in relation to) a physical objectwithin an augmented reality experience. For example, the augmentedreality system can track user interactions with a physical input device(e.g., the user typing on a physical keyboard) to generate new virtualobjects in the augmented reality experience (e.g., a virtual graphicoverlay on a computer screen that includes text corresponding to thetracked typing). In another example, the augmented reality system cantrack user interactions with a physical input device (e.g., a userpushing buttons on a physical game controller) to affect existingvirtual objects in the augmented reality experience (e.g., virtual gamecharacters from a virtual video game).

In at least one embodiment, the augmented reality system can furtherdetect augmented-reality-computing devices in a shared augmented realityexperience within a common physical environment. For example, theaugmented reality system can detect that two separateaugmented-reality-computing devices are generating the same set ofaugmented reality experiences within a common physical environment. Inresponse, the augmented reality system can integrate the augmentedreality experience for both devices in order for those devices to shareinformation. Thus, the users of those augmented-reality-computingdevices can cooperatively work through the same augmented realityexperience within the common physical environment.

As mentioned above, the augmented reality system provides many technicaladvantages and benefits over conventional augmented reality systems andmethods. For example, the augmented reality system improves theefficiency with which conventional augmented reality systems render andpresent augmented reality experiences. In comparison to conventionalsystems, the disclosed augmented reality system more efficiently usesand extends computing resources by selectively rendering or omittingcertain virtual objects from an augmented reality experience andintegrating an analogous physical object from a physical environmentinstead of such virtual objects. The disclosed augmented reality systemcan further extend computing resources by presenting or rendering onlyportions of a virtual object that differ from an analogous physicalobject—thereby avoiding the additional computing resources needed torender a full virtual object. The disclosed augmented reality system canaccordingly use a physical object in conjunction with virtual objects tocreate a more realistic augmented-reality experience. By modifying andgenerating fewer graphical features for a virtual object of an augmentedreality experience based on integrating an analogous physical object,for instance, the augmented reality system saves the computer processingpower and transitory memory that would have conventionally been used byexisting augmented-reality-display devices to render virtual objects forthe same or similar augmented reality experiences. In some cases, theaugmented reality system further saves memory storage that would havebeen utilized in storing (or transferring memory for) three-dimensionalmodels or other information associated with the virtual objects oncerendered.

In addition to more efficient virtual renderings, in some cases, theaugmented reality system improves the efficiency with which systemsgenerate or modify sounds for augmented reality. For example, theaugmented reality system can save computer processing resources byconsolidating one or more audio streams of a complex sound that isanchored to a physical object and then modify the consolidated audiostreams. As explained further below, in one or more embodiments, theaugmented reality system generates these efficiencies in consolidatingor modifying audio streams (and other acoustic sound features) byleveraging the way the human auditory system understands andinterpolates sound, such that a user of the augmented reality systemnotices no decrease in sound quality despite sound modifications thatsave computer processing and memory.

Moreover, the augmented reality system avoids the rigid requirements ofaugmented reality experiences that are typically imposed on conventionalsystems. For example, augmented reality experiences are generallynon-scalable. In some instances, conventional systems require renderingall virtual objects within an augmented reality scene regardless of thephysical environment over which the augmented reality scene or otheraugmented reality experience is overlaid and the functionality of anaugmented-reality-computing device. The augmented reality systemovercomes this rigidity by generating augmented reality experiences thatare scalable based on the contents of the current physical environment.

For example, an augmented reality experience may include a specific typeof virtual speaker corresponding to music for the experience. Byanchoring and modifying sound to a physical speaker from a physicalenvironment rather than to the virtual speaker, the augmented realitysystem can scale down the sound quality or other characteristics for theaugmented reality experience. Similarly, the augmented reality systemcan scale down rendering virtual objects based on the physical objectsdetected in a physical environment. The augmented reality system canfurther utilize more or fewer physical objects within a physicalenvironment depending on the processing and memory capabilities of agiven augmented-reality-computing device. Thus, the augmented realitysystem is more flexible than conventional systems because it can adjustan augmented reality experience to include or exclude virtual objects(or modify sounds) based on the physical objects currently available andcomputing device capabilities.

As illustrated by the foregoing discussion, the present disclosureutilizes a variety of terms to describe features and advantages of theaugmented reality system. Additional detail is now provided regardingthe meaning of such terms. For example, as used herein, “augmentedreality” refers to a composite view including computer-generatedelements real-world or physical elements from a physical environment.For instance, in one or more embodiments, the augmented reality systemgenerates an augmented reality experience including one or more virtualobjects and positions the virtual objects over the user's view within anaugmented-reality-computing device. In one or more embodiments, theaugmented reality system presents and/or renders an augmented realityexperience utilizing a particular physical object instead of ananalogous virtual object. Additionally, in at least one embodiment, theaugmented reality system presents an augmented reality experience bysuperimposing a virtual graphic overlay over a portion of a particularphysical object or over the entire particular physical object.

As used herein, an “augmented-reality-computing device” refers to acomputing device that generates and presents an augmented realityexperience. For example, an augmented-reality-computing device cangenerate, render, and/or present a display of an augmented realityexperience comprising one or more virtual objects and physical objects.Additionally or alternatively, an augmented-reality-computing device cangenerate and/or present an audio-only augmented reality experiencewithout virtual objects as visual components, but rather generate orpresent one or more virtual sounds. An augmented-reality-computingdevice can be a head-mounted-computing device, such as a virtual realityheadset, mixed reality headset, augmented reality glasses, smartglasses, and/or a head-embedded computing device. In some cases, othercomputing devices can also function as augmented-reality-computingdevices, such as smart phones and/or smart tables (e.g., withrear-facing cameras). In at least one embodiment, anaugmented-reality-computing device also includes audio playback features(e.g., headphones, ear buds) that provide audio associated with anaugmented reality experience to the user wearing the device. Anaugmented-reality-computing device can further include variousenvironmental sensors (e.g., a gyroscope, an accelerometer) to enablemovement detection.

As further used herein, an “augmented reality experience” refers to oneor more augmented reality graphics, sounds, or other features generatedor provided via an augmented-reality-computing device. Such features canbe part of a game experience, an educational experience, a businessexperience, an entertainment experience, or similar. In one or moreembodiments, an augmented reality experience includes one or moreaugmented reality scenes, each including virtual objects and/or soundsassociated with each augmented reality scene. Accordingly, as describedbelow, this disclosure's references to augmented reality experience maycomprise or constitute an augmented reality scene with one or morevirtual objects. By contrast, in some cases, an augmented realityexperience includes generating or presenting one or more virtual soundswithout rendering or otherwise presenting virtual objects.

As just indicated, an augmented reality experience can include anaugmented reality scene. An “augmented reality scene” refers to acomposite image or view comprising one or more virtual objects andphysical (or real-world) objects. In some cases, an augmented realityscene comprises a three-dimensional image or environment comprising botha virtual object and a physical object with which a user can interactusing computer detection or environmental sensors. In one or moreembodiments, an augmented reality scene further includes or correspondsto one or more sounds that further inform or enhance the augmentedreality scene. For example, a sound for an augmented reality scene caninclude music, sound effects, human speech, and any other type of sound.

As used herein, a “virtual object” refers to acomputer-generated-graphical object rendered as part of an augmentedreality scene or other augmented reality experience. For example, avirtual object may include an object generated by a computing device fordisplay within an augmented reality scene or for use within an augmentedreality application. Such virtual objects may be, but are not limitedto, virtual accessories, animals, books, electronic devices, vehicles,windows, or any other graphical object created by a computer. A virtualobject can have features, characteristics, and other qualities (e.g., asdefined by a model, a file, a database).

As used herein, an “analogous virtual object” is a virtual object for anaugmented reality experience that is determined to be an analog of acorresponding physical object in a physical environment. For example, ananalogous virtual object may or may not be identical to a correspondingphysical object. In at least one embodiment, an analogous virtual objectshares at least one feature and/or characteristic of a correspondingphysical object.

In one or more embodiments, the augmented reality system can modifyacoustic features based on spectral localization cues. As discussedbelow, “spectral localization cues” refer cues that inform or stimulatehow the human brain localizes sound outside of the human head. Spectrallocalization cues are generally individual to a user and include how theuser's head and the intricacies of his or her ears effect thefrequencies that eventually reach the user's ear drums. For example, dueto the complexities of the human ear (e.g., the shape of the outer earwith its concave and asymmetrical folds), a person may only hear asubset of the spectrum of frequencies within a single sound. A differentuser may hear a different subset of spectrum of frequencies within thesame sound because of physical differences in his or her ears. The waythat the person hears and locates sound is further affected by the sizeand shape of his or her head, which sound must travel around to reachboth ears. In at least one embodiment, the augmented reality systemutilizes average spectral localization cues (e.g., based on an averageear and head size) to modify the originating location of a sound.

As used herein, a “physical environment” refers to a physical spacesurrounding or within proximity to a user detected in whole or in partyby an augmented-reality-computing device. In some embodiments, aphysical environment includes physical objects located in a physicalspace detected by a camera, microphone, or other sensor of anaugmented-reality-computing device. A physical environment can beindoors (e.g., a bedroom, and office, a classroom) or outdoors (e.g., apark, a beach, a playground, a shopping mall). A physical environmentcan include area indicators (e.g., a floor, walls, a ceiling), whichdefine the area or confines of the physical environment, and physicalobjects, which reside within the defined area or confines of thephysical environment.

As used herein, a “physical object” refers to a real-world article in aphysical area. Such physical objects may be, but are not limited to,physical accessories, animals, books, electronic devices, vehicles,windows, or any other tangible or physical object in the real world. Insome cases, physical objects may be free-standing or may be positionedon other physical objects (e.g., as a lamp may be positioned on desk).Physical objects can have classifications, types, features, andcharacteristics, as discussed below.

As used herein, “acoustic features” refers to sound components presentin (or detected from) a sound. For example, acoustic features of a soundmay include an amplitude for the sound, one or more frequencies thatmake up the sound, the volume of the sound, timbre of the sound, thereverberation of the sound, or the color or loudness of the sound. In atleast one embodiment, acoustic features of a complex sound may includetwo or more audio streams that represent sub-sounds within the complexsound.

As used herein, a “sound profile” refers to acoustic instructionsassociated with a virtual object or other object. For example, a soundprofile associated with a virtual object can inform how soundsoriginating from the virtual object should sound. As such, the virtualobject's sound profile may include various acoustic features, such as asound volume, a level of sound degradation, a level of soundenhancement, and various level specifications (e.g., associated withtreble levels, bass levels).

As used herein, a “data stream” refers to a sequence of data captured byan augmented-reality-computing device. In one or more embodiments, adata stream can include an image stream captured by a camera or otherimage-capturing device, an audio stream captured by a microphone orother audio input, or a data stream captured by one or moreenvironmental sensors associated with the augmented-reality-computingdevice. For example, a data stream may include optical data captured byan optical sensor or laser data captured by a laser scanner. In eithercase, the data stream may be captured as part of a simultaneous locationand mapping (“SLAM”). As a further example, an environmental data streamfrom a gyroscope of an augmented-reality-computing device can include astream of data indicating a real-time tilt and orientation associatedwith the augmented-reality-computing device. A data stream may becontinuous or intermittent or have a starting point or capture andending point of capture. For example, an augmented-reality-computingdevice may capture one or more intermittent sequences of data inresponse to detecting movement (e.g., while the user is moving his orher head), and then return to a passive mode where data sequences are nolonger captured. Alternatively, an augmented-reality-computing devicecan capture one or more data streams continuously.

As noted above, a data stream may include an image stream or an audiostream. As used herein, an “image stream” refers to a sequence of imagescaptured by (or received from) a camera or other image-capturing device.In some case, an image stream includes a sequence of still imagescaptured by a camera divide (e.g., a micro-camera associated with anaugmented-reality-computing device). In at least one embodiment, animage stream can be provided by a camera in real time or near-real time.Additionally, as used herein, an “audio stream” refers to a sequence ofdata comprising audio information. In some cases, an audio streamincludes a sequence of data captured by a microphone that a computingdevice encodes or transforms into data packets comprising audioinformation (e.g., acoustic tones and/or frequencies).

FIG. 1 illustrates an example block diagram of an environment 100 forimplementing an augmented reality system 102. As illustrated in FIG. 1 ,the augmented reality system 102 includes augmented-reality-computingdevices 106 a and 106 b, and server(s) 104, which are communicativelycoupled through a network 110. As shown in FIG. 1 , theaugmented-reality-computing devices 106 a and 106 b include augmentedreality applications 108 a and 108 b, respectively. Additionally shownin FIG. 1 , the server(s) 104 includes an augmented reality system 102.Further shown in FIG. 1 , the augmented-reality-computing devices 106 aand 106 b are associated with users 112 a and 112 b, respectively.

The augmented-reality-computing devices 106 a and 106 b, and theserver(s) 104 communicate via the network 110, which may include one ormore networks and may use one or more communication platforms ortechnologies suitable for transmitting data and/or communicationsignals. In one or more embodiments, the network 110 includes theInternet or World Wide Web. The network 110, however, can includevarious other types of networks that use various communicationtechnologies and protocols, such as a corporate intranet, a virtualprivate network (“VPN”), a local area network (“LAN”), a wireless localnetwork (“WLAN”), a cellular network, a wide area network (“WAN”), ametropolitan area network (“MAN”), or a combination of two or more suchnetworks.

Although FIG. 1 illustrates a particular arrangement of theaugmented-reality-computing devices 106 a and 106 b, the server(s) 104,and the network 110, various additional arrangements are possible. Forexample, the augmented-reality-computing devices 106 a and 106 b maydirectly communicate with the augmented reality system 102, bypassingthe network 110. Further, the environment 100 can include any number ofaugmented-reality-computing devices communicating with the augmentedreality system 102. Additional details relating to the network 110 areexplained below with reference to FIG. 10 .

Although FIG. 1 illustrates the augmented reality system 102 hosted bythe server(s) 104, the functionality of the augmented reality system 102may reside elsewhere. For example, some or all of the functionality ofthe augmented reality system 102 may be performed by the augmentedreality applications 108 a and 108 b on the augmented-reality-computingdevices 106 a and 106 b, respectively. Thus, theaugmented-reality-computing devices 106 a and 106 b can generate anddisplay or otherwise present augmented reality experiences in theabsence of a network connection to the augmented reality system 102.Additionally or alternatively, the augmented-reality-computing devices106 a and 106 b can provide an image stream of a physical environment tothe augmented reality system 102 via the network 110, and then receiveand display data for an augmented reality experience generated by theaugmented reality system 102. Additionally or alternatively, theaugmented-reality-computing devices 106 a and 106 b may receive datacomprising computer-executable rendering instructions from the augmentedreality system 102 and generate a rendering of an augmented realityexperience based on the rendering instructions.

As suggested above, the augmented-reality-computing devices 106 a and106 b each include an augmented reality display, a video capturingdevice (e.g., a digital camera), and an audio playback mechanism (e.g.,headphones). For example, in one or more embodiments, the augmentedreality display of the augmented-reality-computing devices 106 a and 106b displays a virtual graphic overlay displayed in connection with thewearer's normal view. In at least one embodiment, the augmented realitydisplay operates as a pair of lenses (e.g., eye glass lenses, contactlenses) positioned over the wearer's eyes. Additionally, in one or moreembodiments, the video capturing devices associated with theaugmented-reality-computing devices 106 a and 106 b are micro digitalvideo cameras mounted (e.g., to an earpiece, or over the bridge of thewearer's nose) to the augmented-reality-computing devices 106 a and 106b, respectively. Further, the audio playback mechanism of theaugmented-reality-computing devices 106 a and 106 b may include rightand left headphones, ear buds, or speakers built into a portion of theaugmented-reality-computing devices 106 a and 106 b (e.g., built intothe earpieces). Thus, in some embodiments, theaugmented-reality-computing devices 106 a and 106 b are similar toeyeglasses with all the component parts built-in. In one or moreembodiments, the augmented-reality-computing devices 106 a and 106 balso include at least one processor capable of executing software code.

As mentioned above, in some embodiments, the augmented reality system102 anchors acoustic or graphical features of an augmented realityexperience to a physical object in a physical environment based on thephysical object being analogous to a virtual object in the augmentedreality experience. More specifically, the augmented reality system 102can render or otherwise present the augmented reality experience withoututilizing the analogous virtual object, but rather utilizing thephysical object. In accordance with one or more embodiments, FIG. 2illustrates an overview of the augmented reality system 102 determininga physical object from a physical environment corresponds to ananalogous virtual object for an augmented reality experience andpresenting the augmented reality experience by modifying one or morefeatures of the experience to integrate the physical object.

As depicted, FIG. 2 illustrates the augmented reality system 102capturing a data stream from an augmented-reality-computing device 202.In one or more embodiments, the augmented reality system 102 can capturean image stream via a camera of the augmented-reality-computing device.The augmented reality system 102 can receive the image steam over anetwork connection with the augmented-reality-computing device.Additionally or alternatively, the augmented reality system 102 cancapture and process the image stream from the camera at theaugmented-reality-computing device.

The augmented reality system 102 can further determine a physical objectcorresponds to an analogous virtual object 204. More specifically, theaugmented reality system 102 can determine that a physical object in thephysical environment corresponds to an analogous virtual object in theaugmented reality experience. In one or more embodiments, the augmentedreality system 102 makes this determination in part by mapping thephysical environment to identify the physical objects in the physicalenvironment. For example, the augmented reality system 102 can utilizeor implement a SLAM system to extract area indicators (e.g., walls,floor, ceiling) and objects (e.g., windows, furniture, books, dishes,toys, TVs) of the physical environment, determine a location of theaugmented-reality-computing device within the physical environment, andcalculate distances (e.g., horizontal, vertical, and angular) betweenthe augmented-reality-computing device and the extracted area indicatorsand objects.

In one or more embodiments, the augmented reality system 102 furtherinventories the physical objects in the physical environment. Forexample, the augmented reality system 102 can utilize image analysis,web-lookups, and other techniques to identify and classify the physicalenvironment objects.

For instance, utilizing any of these techniques, the augmented realitysystem 102 determines (i) that a particular shape or outline in thephysical environment is an object and (ii) a category or classificationassociated with the object based on broad categories or classifications,such as “furniture,” “book,” “décor.” Based on the broad classificationof the object, the augmented reality system 102 can further determineadditional features and characteristics of the object, such as thefunctionality of the object, the physical limitations of the object, andso forth. In at least one embodiment, the augmented reality system 102can store all this information in association with the identifiedphysical object for later use in generating and presenting an augmentedreality experience.

Similarly, the augmented reality system 102 can inventory virtualobjects associated with an augmented reality experience. For example,the augmented reality system 102 can access an augmented reality sceneof an augmented reality experience to determine one or more virtualobjects associated with the augmented reality scene. In one or moreembodiments, the augmented reality system 102 can analyze metadata,display instructions, and other information associated with theaugmented reality scene to identify virtual objects included in theaugmented reality scene. The augmented reality system 102 can furtheridentify a type or classification of the virtual objects based on imageanalysis, metadata, or other display instructions associated with theaugmented reality scene. Based on the identified type or classification,the augmented reality system 102 can further determine features andcharacteristics of the virtual objects.

The augmented reality system 102 can then determine that one or morephysical objects of the physical environment correspond to one or morevirtual objects of the augmented reality experience based on theidentified characteristics and features of both the virtual objects ofthe augmented reality experience and the physical objects of thephysical environment. For example, the augmented reality system 102 cancalculate an object-matching score between each physical object and eachvirtual object indicating a degree to which one or more features orcharacteristics of each physical object match one or more features orcharacteristics of each virtual object.

Briefly, in some cases, the augmented reality system 102 can calculatethe object-matching score between a physical object and a virtual objectby adding a point or value to the object-matching score for eachmatching characteristic and/or feature identified between the twoobjects. In at least one embodiment, the augmented reality system 102can further weight the point or value based on a relevancy associatedwith the matching characteristic and/or feature (e.g., as with acharacteristic and/or feature indicating appearance or function).

In one or more embodiments, the augmented reality system 102 determinesthat a particular physical object corresponds to a particular virtualobject when the object-matching score between the two objects satisfiesan object-matching threshold. If the object-matching score associatedwith a particular physical object and a particular virtual objectsatisfies the object-matching threshold, the augmented reality system102 can determine that the physical object corresponds to the analogousvirtual object.

As further shown in FIG. 2 , the augmented reality system 102 cangenerate or modify various acoustic or graphical features for theaugmented reality experience based on the physical object correspondingto the analogous virtual object. For example, in one or moreembodiments, the augmented reality system 102 can modify one or moreacoustic features of a sound for the augmented reality experience tosimulate that the sound originates from the physical object 206. In atleast one embodiment, the augmented reality system 102 can modify theacoustic features of the sound based on horizontal, vertical, andangular distances between the location of the physical object and theaugmented-reality-computing device, as well as on other spectrallocalization cues. The augmented reality system 102 can further modifythe acoustic features based on characteristics of the physical object.In some cases, the overall effect of modifying the acoustic features isto simulate, from the perspective of the user of theaugmented-reality-computing device, that the sound originates from thephysical object, even though the physical object is not actuallyproducing the sound.

In additional or alternative embodiments, the augmented reality system102 can generate or modify virtual objects based on tracking userinteractions with the physical object 208. For example, the augmentedreality system 102 can track user interactions with the physical objectas part of a game or other augmented reality experience. To illustratebut one example, the augmented reality system 102 can track userinteractions with a physical remote control to change the televisionchannel displayed on a virtual television screen within the augmentedreality experience.

In an additional or alternative embodiment, the augmented reality system102 can generate a virtual graphic overlay associated with the virtualobject 210. For example, the augmented reality system 102 can generatethe virtual graphic overlay based on the analogous virtual object tocover or obscure all or a portion of the physical object when thevirtual graphic overlay is positioned at the location of the physicalobject. The augmented reality system 102 can generate the virtualgraphic overlay based on visual characteristics of the analogous virtualobject, such that the overlay causes the physical object to appeardifferently to the user of the augmented-reality-computing device.Additionally or alternatively, the augmented reality system 102 cangenerate the virtual graphical overlay based on a difference between thephysical object and the analogous virtual object by rendering only aportion of a virtual object that differs from an analogous physicalobject. In at least one embodiment, the augmented reality system 102 canupdate or replace the virtual graphic overlay based on further userinteractions with an area of the physical object on which the virtualgraphic overlay is superimposed.

As further shown in FIG. 2 , the augmented reality system 102 canfurther present the augmented reality experience without the analogousvirtual object 212. As suggested above, in some cases, the augmentedreality system 102 presents the augmented reality experience using thephysical object rather than an analogous virtual object. For example, inresponse to determining the object-matching score between the twoobjects is greater than or equal to the object-matching threshold, theaugmented reality system 102 can determine that the virtual object isanalogous to the physical object and anchor one or more features of theaugmented reality scene to the physical object rather than rendering theanalogous virtual object. For instance, the augmented reality system 102can associate the one or more features of the augmented realityexperience with the location of the physical object, as determined viaSLAM or a similar algorithm. In at least one embodiment, the augmentedreality system 102 can present the augmented reality experience fordisplay via the augmented-reality-computing device without the analogousvirtual object. In some cases, the augmented reality system 102 canpresent the augmented reality experience comprising virtual audio (e.g.,sound effects) via the augmented-reality-computing device without theanalogous virtual object or other virtual objects.

FIGS. 3A and 3B illustrate additional detail with regard to theaugmented reality system 102 rendering an augmented reality experienceand modifying features of the augmented reality experience. For example,FIG. 3A illustrates the augmented reality system 102 determining objectsand locations within a physical environment. FIG. 3B illustrates theaugmented reality system 102 modifying acoustic features of a sound foran augmented reality experience to simulate that the sound originatesfrom a particular physical object in the physical environment.

As shown in FIG. 3A, the user 112 a can view a physical environment 302through a display of the augmented-reality-computing device 106 a. Asfurther shown in FIG. 3A, the physical environment 302 includes physicalobjects 304 a, 304 b, 304 c, 304 d, and other area indicators such as afloor 306, walls 308 a, 308 b, and a ceiling 310. In one or moreembodiments, prior to rendering or providing an augmented realityexperience including virtual objects, the augmented reality system 102maps the physical environment 302 utilizing an image stream captured bythe augmented-reality-computing device 106 a. For example, the augmentedreality system 102 can utilize a mapping protocol, such as SLAM todetermine: (i) the three-dimensional position of theaugmented-reality-computing device 106 a within the physical environment302 and (ii) the spatial relationships between theaugmented-reality-computing device 106 a and the area indicators andobjects in the physical environment 302.

In one or more embodiments, the augmented reality system 102 utilizesenvironmental sensor data to map the physical environment 302 andlocalize the augmented-reality-computing device 106 a. For example, theaugmented reality system 102 can utilize the image stream captured byone or more cameras of the augmented-reality-computing device 106 a.Additionally, the augmented reality system 102 may utilize additionalenvironmental sensor data originating from theaugmented-reality-computing device 106 a including, but not limited to,gyroscopic data, accelerometer data, light sensor data, depth sensordata, and GPS data.

Based on this environmental sensor data, the augmented reality system102 can identify the area indicators of the physical environment 302.For example, the augmented reality system 102 can identify the walls 308a, 308 b, the floor 306, and the ceiling 310 based on an analysis of thecaptured image stream in connection with the additional environmentalsensor data. In one or more embodiments, the augmented reality system102 can further differentiate the physical objects 304 a-304 d from thearea indicators of the physical environment 302. For example, utilizingthe image stream and other environmental sensor information, theaugmented reality system 102 can identify and further classify thephysical objects 304 a-304 d within the physical environment 302.

In one or more embodiments, the augmented reality system 102 identifiesthe physical objects 304 a-304 d by utilizing image analysis techniquesin connection with outlines within the physical environment 302 todetermine an object, object type, and/or object classificationassociated with each outline. For example, the augmented reality system102 can utilize image comparison to find a closest match between an areaof the physical environment 302 (e.g., an outline within the physicalenvironment 302) to a known object. Based on metadata and otherinformation associated with the matched known object, the augmentedreality system 102 can extrapolate that the area within the physicalenvironment 302 is associated with a physical object that has certaincharacteristics and/or features.

The augmented reality system 102 can also determine spatialrelationships between the area indicators of the physical environment302, the physical objects 304 a-304 d within the physical environment302, and the augmented-reality-computing device 106 a. For example, theaugmented reality system 102 can determine distances between theaugmented-reality-computing device 106 a and each of the physicalobjects 304 a-304 d. In one or more embodiments, based on the imagestream and other environmental sensor data, the augmented reality system102 can determine one or more of a vertical distance, a horizontaldistance, and an angular distance between theaugmented-reality-computing device 106 a and each of the physicalobjects 304 a-304 d.

Based on these spatial relationships, the augmented reality system 102can generate a virtual map (e.g., a sparse reconstruction, a dense 3Dpoint cloud) of the physical environment 302 relative to theaugmented-reality-computing device 106 a. For example, the augmentedreality system 102 can generate the map including the locations of thephysical objects 304 a-304 d relative to the augmented-reality-computingdevice 106 a and each other. Based on this map and continued movementtracking associated with the augmented-reality-computing device 106 a,the augmented reality system 102 can maintain accurate positioning ofthe physical objects 304 a-304 d as well as the location of theaugmented-reality-computing device 112 within the physical environment302. In one or more embodiments, the augmented reality system 102utilizes the continually updated location of theaugmented-reality-computing device 106 a within the generatedthree-dimensional map of the physical environment 302 to accuratelyanchor features of an augmented reality experience to one or morephysical objects.

As noted above, FIG. 3B illustrates the augmented reality system 102modifying acoustic features of a sound to simulate the sound originatingfrom a particular physical object in the physical environment. In one ormore embodiments, the augmented reality system 102 modifies acousticfeatures of a sound in a way to stimulate the user 112 a to identify thelocation or origin of the sound in direction and distance. For example,FIG. 3B depicts the user 112 a localizing a sound 312 and the augmentedreality system 102 leveraging this information to successfully “slide”sounds from the perspective of the user 112 a. As discussed below, theuser 112 a localizes the sound 312 based on time and intensitydifferences between both ears, spectral localization cues, and othersignals.

In one or more embodiments, the user 112 a localizes the sound 312 inthree dimensions based on a horizontal angle between the center of thehead of the user 112 a and the source of the sound 312, the verticalangle between the center of the head of the user 112 a and the source ofthe sound 312, and the distance between the center of the head of theuser 112 a and the source of the sound 312. But the way the user 112 ahears the sound 312 is further altered by the head of the user 112 a,which acts as a barrier to change the timbre, intensity, and spectralqualities of the sound 312—further helping the user 112 a determine theorigin of the sound 312.

In at least one embodiment, the augmented reality system 102 quantifiesand represents the way the user 112 a hears the sound 312 using afunction, such as the Head-Related Transfer Function (“HRTF”). Forexample, the Head-Related Transfer Function can be represented as:H _(L) =H _(L)(β,θ,φ,ω,α)=P _(L)(β,θ,φ,ω,α)/P ₀(β,ω)H _(R) =H _(R)(β,θ,φ,ω,a)=P _(R)(β,θ,φ,α)/P ₀(β,ω)Where L and R represent the left ear and right ear, respectively, of theuser 112 a. P_(L) and P_(R) represent the amplitude of sound pressure atthe entrances of the left and right ear canals of the user 112 a. P₀ isthe amplitude of sound pressure at the center of the head of the user112 a (if the user 112 a did not exist). More generally, as illustratedin FIG. 3B, the Head-Related Transfer Functions H_(L), and H_(R) arefunctions of sound source angular position θ, elevation angle φ,distance between the sound source and the center of the head of the user112 a β, the angular velocity ω (if the sound is moving rather thanstationary), and the equivalent dimension of the head of the user 112 aα. Based on these functions, the user 112 a can effectively discern theapproximate location of the source of the sound 312. Note that FIG. 3Billustrates the sound 312 as stationary.

The augmented reality system 102 can exploit the functions by which theuser 112 a hears the sound 312 to simulate the sound 312 originatingfrom a physical object in a physical environment, rather than inside thehead of the user 112 a. As indicated above, sounds appear to come frominside the listener's head unless those sounds are somehow modified. Forexample, the augmented reality system 102 can modify the playbackbalance between left and right headphones of theaugmented-reality-computing device 106 a, and/or the playback volumebetween left and right headphones of the augmented-reality-computingdevice 106 a. The augmented reality system 102 can further angle one ormore playback channels of the left and right headphones of theaugmented-reality-computing device 106 a to alter the amplitude of soundpressure at the entrance to the ears of the user 112 a.

Moreover, the augmented reality system 102 can change the timing ofplayback between the left and right headphones of theaugmented-reality-computing device 106 a to simulate the sound 312originating from a physical object or simulate an effect on the sound312 by the physical object. For example, the human auditory systemutilizes timing differences between when a sound arrives at the left andat the right ear to determine a relative angle from which the soundoriginates. To illustrate, because the sound 312 originates to the rightof the user 112 a, the sound 312 arrives at the right ear of the user112 a before it arrives at the left ear of the user 112 a—due at leastin part to the fact that the sound 312 has to travel around the user'shead. Thus, the augmented reality system 102 can mimic this effect bychanging the timing of when the sound 312 is played out of left andright headphones to effectively fool the user 112 a into thinking thatthe sound 312 originates at an angle outside his or her head.

Additionally, in some embodiments, the augmented reality system 102 canapply a filter to the sound 312 to mimic the localization of the sound312 at the position of the physical object. For example, the augmentedreality system 102 can apply a filter to the sound 312 that changes oneor more levels of the sound 312, that degrades or enhances the sound312, or alters or effects other qualities of the sound 312 to simulatethat the sound originates from the physical object or to simulate aneffect on the sound by the physical object. In any of these ways, theaugmented reality system 102 leverages the ways that the human brainprocesses and understands sound to cause the user 112 a to understandthat the sound 312 originates at and/or is affected by a physicalobject. For example, in some embodiments, the augmented reality systemutilizes work by Facebook Reality Labs in sound propagation to generatespatial audio and allows for volumetric and ambisonic sounds. Additionalinformation related to such work can be found atcreator.oculus.com/learn/spatial-audio/oroculus.com/blog/simulating-dynamic-soundscapes-at-facebook-reality-labs/.

In one or more embodiments, the augmented reality system 102 can furtheraccount for a room impulse response in modifying acoustic features of asound. For example, based on the map of the physical environment, theaugmented reality system 102 can identify and account for echo andreverberation properties of the physical environment when modifyingacoustic features of the sound. To illustrate, the augmented realitysystem 102 can add reverberation to a sound in respond to determiningthat the physical environment is in a six-sided room (e.g., a functionalcube) with a specific size. The augmented reality system 102 may not addreverberation to a sound in response to determining that the physicalenvironment is outside in an area with no walls or large objects off ofwhich a sound would bounce.

In one or more embodiments, the augmented reality system 102 mayconsolidate or reduce a number of audio streams to reduce a number oftimes HRTF is calculated. For example, if a sound of an augmentedreality experience includes multiple audio streams or sound sources. Toillustrates, a car might produce sound from the engine, from themuffler, and from the internal stereo—thereby creating three audiostreams from three sound sources. A conventional system might calculatethe HRTF for each audio stream to further modify the acoustic featuresof the corresponding sounds. The augmented reality system 102, however,leverages the fact that most human hearing is not fine-tuned enough totell the difference between each individual audio streams (e.g.,depending on how far apart the audio streams are from each other).

Based on the HRTF, the augmented reality system 102 can consolidate orreduce two or more of the audio streams without degrading the overallauditory experience for user, while simultaneously generating variouscomputational efficiencies. For example, if a sound of an augmentedreality experience (e.g., the sound of a car) includes three audiostreams (e.g., one for the engine, one for the muffler, one for theinternal stereo), the augmented reality system 102 can combine the audiostreams for the engine and muffler. Thus, to modify various acousticfeatures of the sound, the augmented reality system 102 only needs tocalculate the HRTF for two audio streams rather than three, therebysaving any computing resources that may have been spent in calculatingthe third HRFT.

As discussed above, the augmented reality system 102 can anchor acousticor graphical features of an augmented reality experience to a physicalobject in a physical environment. FIGS. 4A-4D illustrate an example ofthe augmented reality system 102 determining that a physical object inthe physical environment corresponds to an analogous virtual object foran augmented reality experience and anchoring acoustic features of asound from the augmented reality experience to the physical object. Forexample, as shown in FIG. 4A, the user 112 a may be in a physicalenvironment 402. As further shown in FIG. 4A, the physical environment402 includes various physical objects, including a physical object 404.In contrast to FIG. 4A, in some embodiments, FIGS. 4B-4D depict theaugmented reality system 102 both rendering augmented realityexperiences for display on the augmented-reality-computing device 106 aand generating music (or other sounds) associated with the augmentedreality scenes through headphones connected to theaugmented-reality-computing device 106 a. Alternatively, FIGS. 4B-4D candepict the augmented reality system 102 presenting augmented realityaudio-only experiences (e.g., as in FIG. 4B), and combined audio andvisual experiences (e.g., as in FIGS. 4C and 4D) via theaugmented-reality-computing device 106 a.

As shown in FIG. 4B, for instance, the user 112 a can wear theaugmented-reality-computing device 106 a. In one or more embodiments, asdiscussed above, the augmented-reality-computing device 106 may includeone or more micro-cameras, gyroscopes, accelerometers, processors,headphones, speakers, microphones, and so forth. In response to the useractivating the augmented-reality-computing device 106 a and/or selectinga particular augmented reality experience (e.g., an experience thatenables the user to listen to music), the augmented reality system 102can capture and utilize an image stream and other environmental sensordata from the augmented-reality-computing device 106 a to map thephysical environment 402. The augmented reality system 102 can furtherutilize the generated map to determine the relative position of theaugmented-reality-computing device 106 a to physical objects. Forexample, as discussed above, the augmented reality system 102 canutilize SLAM to determine the location of theaugmented-reality-computing device 106 a, and the horizontal, vertical,and angular distance between the augmented-reality-computing device 106a and the physical object 404.

The augmented reality system 102 can further identify and classify thephysical object 404. For example, the augmented reality system 102 cananalyze an image frame from the image stream captured by theaugmented-reality-computing device 106 a to determine that the physicalobject 404 is a smart speaker utilizing a wireless protocol. Based onidentifying the physical object 404 as a smart speaker, the augmentedreality system 102 can further utilize web lookups, database lookups,and other info to determine features and characteristics associated withthe physical object 404. For example, the augmented reality system 102can determine that the physical object 404 can play audio based on datatransmitted via a wireless protocol and the physical object 404 has aparticular size. In some embodiments, the augmented-reality-computingdevice 106 a detects a wireless broadcast signal from the physicalobject 404, such as a BLUETOOTH broadcast signal.

As indicated above, the augmented reality system 102 can determine anobject-matching score indicating a degree to which one or more of thefeatures or characteristics of the physical object 404 match those ofvarious virtual objects in an augmented reality experience. For example,in response to detecting the user 112 a selecting an augmented realityexperience that includes music, the augmented reality system 102 canfurther identify the virtual objects corresponding to the augmentedreality experience for the music-listening augmented reality experience.In some embodiments, the augmented reality system 102 determines that aselected augmented reality experience includes virtual objects thatmatch a particular music (e.g., virtual object for a music video orvideo game). In other embodiments, the augmented reality system 102determines that an augmented reality experience associated with themusic-listening augmented reality experience includes a single virtualobject—such as a 1990's era virtual stereo that plays the musiccorresponding to the augmented reality experience.

As a further example, in some embodiments, in response to detecting theuser 112 a selecting an augmented reality experience that includes onlymusic, the augmented reality system 102 can utilize the physical object404 based on determining that the characteristics of the physical object404 (e.g., produces audio) match characteristics of the audio-onlyaugmented reality experience. In one or more embodiments, the augmentedreality system 102 can utilize the physical object 404 by anchoringsounds of the audio-only augmented reality experience to the physicalobject 404. For example, the augmented reality system 102 can anchor orassociate acoustic features of a sound of the audio-only augmentedreality experience with a location of the physical object 404 relativeto the augmented-reality-computing device 106 a. In one or moreembodiments, the augmented reality system 102 can store this associationin connection with the augmented reality experience until the anchoredfeatures are triggered or required within the augmented realityexperience.

For example, in response to determining that a sound (e.g., musicplayback) associated with the augmented reality experience should beheard by the user 112 a via the augmented-reality-computing device 106 a(e.g., in response to the user 112 a selecting a “play” optionassociated with the augmented reality experience), the augmented realitysystem 102 can modify the anchored acoustic features to simulate thatthe sound originates from the physical object 404. In one or moreembodiments, the augmented reality system 102 can modify the acousticfeatures of the sound based on the location of the physical object 404relative to the augmented-reality-computing device 106 a.

For instance, as discussed above, the augmented reality system 102 canmodify the acoustic features of the sound based on (i) the distancebetween the location of the physical object 404 and theaugmented-reality-computing device 106 a, (ii) other spectrallocalization cues associated with the location of the physical object404, and (iii) any visual characteristics of the physical object 404(e.g., the size of the physical object 404 a, the direction the physicalobject 404 a is pointed). In at least one embodiment, the augmentedreality system 102 can modify the acoustic features of the sound basedon these considerations such that the amplitude of sound pressureinteracting with the ears of the user 112 a causes the user 112 a tothink that the sound of the augmented reality experience is originatingfrom the physical object 404. The augmented reality system 102 canlikewise modify acoustic features as described in this paragraph when anaugmented reality experience comprises virtual objects.

In one or more embodiments, the augmented reality system 102 can utilizemetadata associated with the augmented reality experience, alone or inconnection with image analysis of an image of the virtual stereo, toidentify features and characteristics of the virtual stereo. Forexample, the augmented reality system 102 can determine that thecharacteristics of the virtual stereo include that the virtual stereocan play sounds, and that the portable stereo has a particular size,shape, and appearance. If the augmented reality experience includesadditional virtual objects, the augmented reality system 102 can repeatthis process for each virtual object associated with the augmentedreality experience.

In at least one embodiment, the augmented reality system 102 calculatesobject-matching scores based on the features or characteristics of thephysical object 404 and the identified features or characteristics ofeach virtual object in the augmented reality experience. For example,the augmented reality system 102 can calculate the object-matching scorebetween the physical object 404 and the virtual stereo indicating adegree to which characteristics or features of the physical object 404match those of the virtual stereo. For instance, the augmented realitysystem 102 can calculate the object-matching score for the physicalobject 404 and the virtual stereo by adding an amount or point to theobject-matching score for each identified match between the features orcharacteristics of the physical object 404 and features orcharacteristics of the virtual stereo.

In one or more embodiments, the augmented reality system 102 can furtherweight the amount or point added to the score based on a relevance of afeature that matches between the two objects. For example, if thematched feature goes to the functionality of the objects (e.g., as withthe feature indicating that both objects play sounds), the augmentedreality system 102 can add an extra weight to the amount or point addedto the object-match score for the physical object 404 a and the virtualstereo.

In one or more embodiments, the augmented reality system 102 candetermine that the virtual stereo in the augmented reality experience isanalogous to the physical object 404 based on the object-matching score.For example, the augmented reality system 102 can determine that thevirtual stereo represents the virtual object associated with the highestcalculated object-matching score is analogous to the physical object404. In the current example, the augmented reality system 102 candetermine that the virtual stereo is analogous to the physical object404 based on the object-matching score between the two objects being thehighest score calculated in connection with the virtual objects in theaugmented reality experience. As further indicated above, in someembodiments, the augmented reality system 102 determines that thevirtual object is analogous to the physical object 404 based on theobject-matching score between the two objects satisfying anobject-matching threshold.

For example, the object-matching threshold for the current augmentedreality experience may be an object-matching score of 5. The augmentedreality system 102 may calculate an object-matching score between thephysical object 404 and the virtual stereo of the augmented realityexperience to be at least 5 based on various weighted and unweightedfeature matches. For instance, the augmented reality system 102 maydetermine that certain appearance features match between the two objectsbecause both objects have speaker covers or grills and playback buttons.The augmented reality system 102 may further determine that there is afunctionality match between the two objects because both include speakercones and gaskets for producing sound. The augmented reality system 102may further weight either or both of these matches because they arerelated to the relevancy of both objects. Accordingly, because theresulting object-matching score satisfies the object-matching threshold,the augmented reality system 102 can determine the virtual stereo isanalogous to the physical object 404 in the physical environment 402.

Returning to FIG. 4B, in response to determining that the virtual stereois analogous to the physical object 404, the augmented reality system102 can anchor one or more acoustic or graphical features of theaugmented reality experience to the physical object 404. For example,and as discussed above, the augmented reality system 102 can anchor orassociate acoustic features of a sound of the augmented realityexperience with a location of the physical object 404 relative to theaugmented-reality-computing device 106 a. In one or more embodiments,the augmented reality system 102 can store this association inconnection with the augmented reality experience until the anchoredfeatures are triggered or required within the augmented realityexperience.

In one or more embodiments, the augmented reality system 102 can alsomodify the anchored acoustic features to further save computingresources associated with the augmented-reality-computing device 106 a.As discussed above, objects can create complex sounds that includemultiple audio streams, such as an engine that generates differentsounds from fan blades, belts, or pistons. In one or more embodiments,the augmented reality system 102 can modify one or more of the audiostreams of a complex sound by degrading, softening, or silencing one ormore of the audio streams.

Additionally or alternatively, the augmented reality system 102 canconsolidate or reduce two or more of the audio streams to further savecomputing resources. In at least one embodiment, the augmented realitysystem 102 can modify or consolidate the audio streams such that thesound, as heard by the user 112 a, is not diminished. For example, asdiscussed above, if a sound of an augmented reality experience (e.g.,the sound of a car) includes three audio streams (e.g., one for theengine, one for the muffler, one for the internal stereo), the augmentedreality system 102 can combine the audio streams for the engine andmuffler. The human auditory system is generally not fine-tuned enough todetermine any loss of audio quality based on this consolidation of audiostreams.

Thus, as shown in FIG. 4B, the augmented reality system 102 can anchoracoustic features of music playback in the augmented reality experienceto the physical object 404. When the music playback is triggered,requested, or otherwise initiated, the augmented reality system 102modifies the acoustic features of the music playback to simulate thatthe music playback originates from the physical object 404. Accordingly,from the perspective of the user 112 a, the smart speaker physicalobject 404 is the source of the music playback within the augmentedreality experience, even though the physical object 404 is not makingany sound within the physical environment 402.

In one or more embodiments, the augmented reality system 102 can anchorgraphical features of the augmented reality experience to the physicalobject 404. For example, as shown in FIG. 4C, in response to determiningthat the virtual stereo is analogous to the physical object 404, theaugmented reality system 102 can generate a virtual graphic overlay 406.The augmented reality system 102 can further render the virtual graphicoverlay 406 within the augmented reality experience at a positionrelative to the augmented-reality-computing device 106 a such that thephysical object 404 is partially or totally covered or obscured by thevirtual graphic overlay 406.

For example, in response to determining that the virtual stereo isanalogous to the physical object 404, the augmented reality system 102can identify one or more visual characteristics of the virtual stereo.More specifically, the augmented reality system 102 can identify visualcharacteristics that indicate a size, a color, an appearance, a surfacetexture, and/or other visual characteristics of the virtual stereo.Utilizing the identified visual characteristics, the augmented realitysystem 102 can generate the virtual graphic overlay 406. In at least oneembodiment, the augmented reality system 102 can then overlay thephysical object 404 with the generated virtual graphic overlay 406. Asshown in FIG. 4C, the augmented reality system 102 can render thevirtual graphic overlay 406 such that the physical object 404 iscompletely obscured from the user 112 a via theaugmented-reality-computing device 106 a.

In one or more embodiments, the augmented reality system 102 can furthermodify the anchored acoustic features of the augmented realityexperience based on features associated with the analogous virtualobject. For example, as shown in FIG. 4D, the augmented reality system102 can identify a sound profile associated with the virtual stereo. Thesound profile indicates a quality of sound and other intricacies of thesound produced by the virtual stereo. To illustrate, the virtual stereomay approximate the appearance and sound of a 1990's era boom box thatplays music with a wide bass range and tinny high notes. In at least oneembodiment, the augmented reality system 102 can identify this soundprofile and modify the acoustic features of music in simulated playbackfrom the physical object 404 to approximate the sound of music playingfrom a 1990's era boom box—rather than from a smart speaker. Thus, theaugmented reality system 102 can degrade the acoustic features of thesound, enhance the acoustic features of the sound, and/or modifyspecific levels (e.g., treble, bass) and/or volumes (e.g., indicated bythe smaller music notes in FIG. 4D) within the acoustic features of thesound to more closely approximate the sound profile associated with thevirtual stereo.

As mentioned above, the augmented reality system 102 can modify anchoredfeatures of an augmented reality experience based on a location or othercharacteristics of a physical object in the physical environment. FIGS.5A and 5B illustrate to the augmented reality system 102 modifyinganchored features of an augmented reality experience based oncharacteristics of a physical object. For example, as shown in FIG. 5A,the user 112 a can be in a physical environment 502 including physicalobjects, such as a physical object 504. As shown in FIG. 5A, thephysical object 504 is a physical or real-world window. In one or moreembodiments, the physical object 504 has various physicalcharacteristics, such as a size, a construction (e.g., including anumber of sashes, casements, mullions, muntins, panes), a configuration(e.g., open or closed), and a thickness. In additional or alternativeembodiments, physical objects can have physical characteristicsincluding, but not limited to, a thickness, a mass, a size, a shape,and/or a density.

As shown in FIG. 5B, after the user 112 a initiates theaugmented-reality-computing device 106 a, the augmented reality system102 can map the physical environment 502 and determine the variouscharacteristics of the physical object 504. For example, the augmentedreality system 102 can determine the physical characteristics of thephysical object 504 utilizing image analysis (e.g., from the imagestream provided by the augmented-reality-computing device 106 a), imagerecognition, database lookups, or other algorithms, as described above.As discussed above, the augmented reality system 102 can furtherdetermine that a virtual object (e.g., a virtual window) in an augmentedreality experience is analogous to the physical object 504. In one ormore embodiments, the augmented reality system 102 can store thecorrespondence between the virtual window and the physical object 504(e.g., physical window), along with the physical characteristics of thephysical object 504 for later use.

As further shown in FIG. 5B, the augmented reality system 102 rendersthe augmented reality experience to include a virtual animal 506 (e.g.,a virtual dinosaur) walking past the virtual or physical window. Inrendering the augmented reality experience for theaugmented-reality-computing device 106 a, the augmented reality system102 can utilize the physical object 504 rather than rendering thevirtual window. In one or more embodiments, the augmented reality system102 can further modify acoustic features of any sound effects (e.g.,dinosaur sound effects) within the augmented reality experience tosimulate that the sound effects are originated from outside the windowphysical object 504—thereby distorting or otherwise muffling the soundeffects.

For instance, depending on a pane thickness and size of the window asthe physical object 504, the augmented reality system 102 can (i)decrease a volume of the sound effects, (ii) reduce one or more specificlevels (e.g., treble, mid-range, bass) of the sound effects, (iii)consolidate or modify various audio streams associated with the dinosaursound effects (e.g., a breathing sound, a foot-fall sound, a mouthopening sound), or (iv) otherwise distort the sound effects. Bydecreasing a volume, reducing a specific sound level, consolidating ormodifying audio streams, the augmented reality system 102 can modify asound for the augmented reality experience to simulate an effect on thesound by the physical object 504. As depicted in FIG. 5B, the augmentedreality system 102 modifies a sound to simulate a filter effect on thesound (e.g., a dinosaur sound) by a window. In at least one embodiment,the augmented reality system 102 can modify the acoustic features of thesound effects based on the physical characteristics of the physicalobject 504 such that, from the perspective of the user 112 a, the soundeffects appear to originate outside the physical object 504, rather thanoriginating at a location of the physical object 504 within the physicalenvironment 502.

FIGS. 6A-6E illustrate additional examples of the augmented realitysystem 102 anchoring features of an augmented reality experience to aphysical object based on a correspondence between the physical objectand an analogous virtual object from the augmented reality experience.For example, in FIG. 6A, an augmented reality experience 602 may includea book as part of an augmented reality experiences. For example, theaugmented reality experience 602 may be from a treasure hunt augmentedreality experience and may include a particular interactive book among acollection of books, where the goal of the experience is to help theuser 112 a identify and interact with a particular book to receive aclue as to the next portion of the treasure hunt.

As indicated by FIG. 6A, the augmented reality system 102 can utilizeSLAM in connection with an underlying physical environment to identifyand classify a physical bookshelf 604 and a physical book 606. Theaugmented reality system 102 can further anchor one or more acousticfeatures of the augmented reality experience 602, such that a sound ofthe augmented reality experience 602 (e.g., music, character speech,drumbeats), appears to originate from a particular book from thephysical bookshelf 604.

In at least one embodiment, the augmented reality system 102 canincrease the volume of the sound of the augmented reality experience 602as the user 112 a moves closer to the particular book. For example, theaugmented reality system 102 can analyze sequential images and otherenvironmental sensor data from the augmented-reality-computing device106 a to determine a speed and direction of movement. The augmentedreality system 102 can further use that speed and direction of movementin connection with the generated virtual map of the underlying physicalenvironment to determine the relative distance between the user 112 aand the particular book on the physical bookshelf 604. For instance, theaugmented reality system 102 can utilize motion tracking algorithms,such as kernel-based tracking and/or contour tracking to determine speedand direction of motion associated with the augmented-reality-computingdevice 106 a.

As further shown in FIG. 6A, when the user 112 a opens the physical book606, the augmented reality system 102 can generate and provide a virtualgraphic overlay 608 to further the augmented reality experience. Forexample, the augmented reality system 102 can generate the virtualgraphic overlay 608 to match or to retexture the physical book 606. Theaugmented reality system 102 can further generate the virtual graphicoverlay 608 to include material specific to the augmented realityexperience.

In one or more embodiments, the augmented reality system 102 can furtheradapt the virtual graphic overlay 608 to physical characteristics of thephysical book 606. For example, the augmented reality system 102 canutilize SLAM to determine a size of the physical book 606 relative tothe amount of augmented reality material that should be provided viainteractions with the physical book 606. For example, if the augmentedreality experience 602 includes providing the user 112 a with fourchapters of material via the book virtual object, the augmented realitysystem 102 can generate the virtual graphic overlay 608 to approximate areading position within the augmented reality material when the user 112a opens the physical book 606.

To illustrate, when the augmented reality system 102 detects the user112 a opening the physical book 606 to a half-way-through readingposition, the augmented reality system 102 can generate the virtualgraphic overlay 608 to display the beginning of chapter three of theaugmented reality materials (e.g., the augmented reality materials thatare half-way through the total amount of augmented reality materials).As the augmented reality system 102 detects the user 112 a continuing toflip through the physical pages of the physical book 606, the augmentedreality system can continue to update or re-render the virtual graphicoverlay 608 to approximate the reading progress of the user 112 athrough the corresponding augmented reality materials.

In one or more embodiments, the augmented reality system 102 can updateor alter an augmented reality experience based on user interactions inconnection with a physical object that corresponds to a virtual objectin an augmented reality experience. For example, as shown in FIG. 6B, anaugmented reality experience including the augmented reality experience610 may include the user 112 a typing input into a computer to furthersome goal of the augmented reality experience. In generating theaugmented reality experience 610, the augmented reality system 102 mayaccordingly utilize SLAM to identify and localize a physical keyboard612. The augmented reality system 102 may further determine that avirtual keyboard associated with the augmented reality experience 610 isanalogous to the physical keyboard 612 and can anchor one or morefeatures of the augmented reality experience based on thisdetermination.

Once the features of the augmented reality experience are anchored tothe physical keyboard 612, the augmented reality system 102 can updateor modify aspects of the augmented reality experience 610 based ondetecting user interactions with the physical keyboard 612. For example,as shown in FIG. 6B, the augmented reality system 102 can utilize motiontracking to detect the user 112 a typing on the physical keyboard 612.Based on the detected user interactions, the augmented reality system102 can generate a virtual graphical overlay (e.g., positioned over aphysical computer monitor) or a virtual object (e.g., a virtualcomputing monitor) including letters or other symbols corresponding tothe detected typing, or can update or modify other virtual objects inthe augmented reality experience 610 based on the detected typing. Thus,the augmented reality system 102 can utilize detected typing on thephysical keyboard 612 to further modify or update the augmented realityexperience 610, even though the physical keyboard 612 is not physicallyconnected to, or otherwise interfaced with, the augmented reality system102 or any other computing system. In additional or alternativeembodiments, the augmented reality system 102 can similarly track userinteractions with other types of physical input devices such as, but notlimited to, game controllers, computer mice, TV remote controllers, andtouch screen displays.

In one or more embodiments, the augmented reality system 102 can furthergenerate or modify acoustic or graphical features of an augmentedreality experience based on a correspondence between a virtual object ofthe augmented reality experience and a physical object of the physicalenvironment. For example, in FIG. 6C, an augmented reality experience614 may comprise an augmented reality scene in which the user 112 a canlisten to music from a record player. In response to identifying aphysical record player 616, the augmented reality system 102 can anchorboth visual and acoustic features of the augmented reality experience614 to the physical record player 616.

The augmented reality system 102 can also determine the sound profileassociated with the physical record player 616 (e.g., the type andquality of music playback of which the physical record player 616 iscapable). The augmented reality system 102 can then play music virtuallyutilizing the physical record player 616. For example, the augmentedreality system 102 can modify the acoustic features (e.g., both thelocalization features and sound quality features) of the music tosimulate that the music is being played by the physical record player616.

Additionally, the augmented reality system 102 anchor graphical featuresof the augmented reality experience 614 to the physical record player616. For example, the augmented reality system 102 can anchor graphicalcharacteristics of the augmented reality experience 614 by generating apartial virtual graphic overlay 618 and positioning the partial virtualgraphic overlay 618 over a portion of the physical record player 616.More specifically, the augmented reality system 102 can generate thepartial virtual graphic overlay 618 to include hovering text indicatinga song title associated with the music of the augmented realityexperience 614, and a record that appears to be spinning on the physicalrecord player 616. Accordingly, in this scenario, the augmented realitysystem 102 generates the augmented reality experience 614 to simulatemusic originating from the physical record player 616 while the physicalrecord player 616 spins a record, even though there is nothing actuallybeing played by the physical record player 616.

Similarly, the augmented reality system 102 can anchor additionalgraphical features of an augmented reality experience to additionalphysical objects in a physical environment. For example, as shown inFIG. 6D, an augmented reality experience may include an augmentedreality experience 620 wherein the user 112 a plays a video game on aphysical game system (that may be disabled). For example, as shown inFIG. 6D, the user 112 a may encounter a physical game console 622 thatmay no longer function (e.g., due to age or disrepair) connected to aphysical screen display 624. In response to determining the type andcapabilities of the physical game console 622, the augmented realitysystem 102 can anchor features of the augmented reality experience 620to both the physical game console 622 and the physical screen display624.

For instance, the augmented reality system 102 can generate and positionthe partial virtual graphic overlay 626, including hovering textdetailing a game title and a portion of game cartridge, on a portion ofthe physical game console 622. The augmented reality system 102 canfurther generate a virtual graphic overlay 628 for a video display andposition the virtual graphic overlay 628 on the physical screen display624 to simulate that the video game is being played by the physical gameconsole 622 and displayed by the physical screen display 624. Asdiscussed above, the augmented reality system 102 can also track userinteractions with a game controller so that the augmented reality system102 can interface with a virtual machine (“VM”) system, or similar, inorder to accurately reflect the game play of the user 112 a within theaugmented reality experience 620.

In one or more embodiments, the augmented reality system 102 can alsogenerate interactive augmented reality experiences between two or moreusers within a physical environment. For example, as shown in FIG. 6E,an augmented reality experience may include users sharing an interactiveexperience with shared augmented reality experiences. As shown in FIG.6E, the users 112 a and 112 b may be located in a physical trainstation. Each of the users 112 a and 112 b may be viewing an augmentedreality experience 630 at the same time. In one or more embodiments,based on both the augmented-reality-computing device 106 a and theaugmented-reality-computing device 106 b being within the samegeographic area, the augmented reality system 102, via theaugmented-reality-computing device 106 a can utilize SLAM, BLUETOOTH,Wi-Fi, or a similar network connection, to detect theaugmented-reality-computing device 106 b, and/or vice versa.

In response to determining that both the augmented-reality-computingdevices 106 a and 106 b are generating the augmented reality experience630 for the same or shared augmented reality experience, the augmentedreality system 102 can generate and position virtual objects as part ofthe augmented reality experience. As shown in FIG. 6E, for example, theaugmented reality system 102 generates a virtual message 634 for displayon the augmented-reality-computing device 106 a to identify the user 112b as a co-user within an augmented reality experience 630. The augmentedreality system 102 can also generate a virtual message 636 for displayon the augmented-reality-computing device 106 b to identify the user 106a as a co-user within the augmented reality experience 630. For example,the augmented reality system 102 can generate the virtual messages 634and 636 either at a central server or through a shared link between theaugmented-reality-computing devices 106 a and 106 b.

Furthermore, the augmented reality system 102 can generate a virtualmessage 638 for display by both the augmented-reality-computing devices106 a and 106 b. In some cases, the virtual message 638 indicates aphysical location or physical object as part of the same or sharedaugmented reality experience. For instance, in some embodiments, theaugmented-reality-computing devices 106 a and 106 b can respectivelydetect interactions by the users 112 a and 112 b with the virtualmessage 638—and generate additional virtual objects—as the users 112 aand 112 b cooperatively work their way through the same or sharedaugmented reality experience.

FIG. 7 illustrates a detailed schematic diagram of an embodiment of theaugmented reality system 102 described above. In one or moreembodiments, the augmented reality system 102 includes a devicecommunicator 702, a map generator 704, an object identifier 706, anobject-matching score generator 708, an anchor generator 710, a ARexperience renderer 712, a feature modifier 714, an overlay generator716, an interaction tracker 718, and a data storage 720 including objectdata 722, physical environment data 724, and augmented realityexperience data 726.

As discussed above, the augmented reality system 102 can be hosted by aserver (e.g., the server(s) 104 as shown in FIG. 1 ) or can reside onany of the augmented-reality-computing devices 106 a and 106 b. Forexample, if hosted by a server, the augmented reality system 102 cancommunicate with the augmented-reality-computing devices 106 a and 106 bto receive image streams and other environmental sensor data, and toprovide renderings or rendering instructions for augmented realityexperiences including virtual objects. If the augmented reality system102 is contained by the augmented-reality-computing device 106 a, thefunctionality of the augmented reality system 102 may be whollycontained by the augmented-reality-computing device 106 a. Additionallyor alternatively, the parts of the functionality of the augmentedreality system 102 may be hosted by a server, while other parts of thefunctionality of the augmented reality system 102 may be performed bythe augmented-reality-computing device 106 a.

As shown in FIG. 7 , and as mentioned above, the augmented realitysystem 102 can include the device communicator 702. In one or moreembodiments, the device communicator 702 handles communications betweenthe augmented reality system 102 and the augmented-reality-computingdevice 106 a—if the augmented reality system 102 is not located on theaugmented-reality-computing device 106 a. For example, the devicecommunicator 702 can capture an image stream of a physical environmentfrom the augmented-reality-computing device 106 a. The devicecommunicator 702 can also receive environmental sensor information fromthe augmented-reality-computing device 106 a indicating a position,location, movement, etc. of the augmented-reality-computing device 106a. The device communicator 702 can further provide augmented realityexperiences and/or rendering instructions for augmented realityexperiences to the augmented-reality-computing device 106 a.

Additionally, the device communicator 702 can handle communicationsbetween two or more augmented-reality-computing devices 106 a and 106 b.For example, in a scenario where two users 112 a and 112 b arecooperating within an augmented reality experience toward a common goal,the device communicator 702 can communicate information between theaugmented-reality-computing devices 106 a and 106 b. In one or moreembodiments, the device communicator 702 can communicate positionalinformation, image stream information, and other environmental sensorinformation between the augmented-reality-computing devices 106 a and106 b.

As shown in FIG. 7 , and as mentioned above, the augmented realitysystem 102 also includes the map generator 704. In one or moreembodiments, the map generator 704 utilizes the SLAM system, or anyother appropriate mapping system, to map a physical environment relativeto the augmented-reality-computing device 106 a. For example, the mapgenerator 704 can utilize SLAM to extract features of the physicalenvironment and determine objects within the physical environment. Themap generator 704 can further determine relative distances and anglesbetween the features and objects of the physical environment and theaugmented-reality-computing device 106 a. Based on all this information,the map generator 704 can generate a three-dimensional map of thephysical environment and localize the augmented-reality-computing device106 a within the physical environment.

In one or more embodiments, the map generator 704 can further update thelocation of the augmented-reality-computing device 106 a. For example,based on movement signals received from the augmented-reality-computingdevice 106 a (e.g., from a gyroscope, an accelerometer, an imagestream), the map generator 704 can calculate an updated location of theaugmented-reality-computing device 106 a within the three-dimensionalmap of the physical environment. The map generator 704 can furtherupdate the relative distances and locations of the physical objects fromthe augmented-reality-computing device 106 a based on the movementsignals.

As mentioned above, and as shown in FIG. 7 , the augmented realitysystem 102 includes the object identifier 706. In one or moreembodiments, the object identifier 706 identifies the one or morephysical objects within a physical environment. For example, the objectidentifier 706 can receive an indication of a physical object from themap generator 704 and can utilize image analysis and other detectionmethods to determine what the indicated physical object actually is. Inat least one embodiment, the object identifier 706 can utilize heatmaps, machine learning, image comparison, or any other suitabletechnique to identify physical objects in the physical environment.

In one or more embodiments, the object identifier 706 further determinesa type or classification for each identified physical object. Forexample, if the object identifier 706 determines that a physical objectis a lamp, the object identifier 706 can further determine that the lampcan be classified as furniture, as décor, as living room furniture, etc.In at least one embodiment, the object identifier 706 can determine thetype or classification of an identified physical object based on a weblookup, a database lookup, machine learning, or other data repositorytechniques.

In response to determining the type or classification of the physicalobject, the object identifier 706 can further determine features andcharacteristics of the physical object. For example, in response toidentifying the lamp and determining that it is furniture, the objectidentifier 706 can further determine that features and characteristicsof the lamp include that it is stationary, that it emits light whenturned on, that it has a specific size, that certain interaction (e.g.,being switched on and off) effect its appearance, and so forth. In atleast one embodiment, the object identifier 706 can determine thefeatures and characteristics of the physical object based on machinelearning, data lookups, or any other appropriate technique.

In one or more embodiments, the object identifier 706 can similarlyidentify and classify virtual objects in an augmented realityexperience. For example, in response to a selection or other indicationof an augmented reality experience (e.g., as selected by the user 112 a,or as dictated by the present augmented reality experience), the objectidentifier 706 can retrieve or otherwise identify the virtual objectsrequired by the augmented reality experience. For instance, the objectidentifier 706 can retrieve the required virtual objects asthree-dimensional image files or other virtual object models from alocation included in rendering instructions associated with theaugmented reality experience. For each identified virtual object, theobject identifier 706 can utilize machine learning, data lookups, imageanalysis, or any other appropriate technique to determine the featuresand characteristics of the virtual object.

As shown in FIG. 7 , and as mentioned above, the augmented realitysystem 102 includes the object-matching score generator 708. In one ormore embodiments, the object-matching score generator 708 calculates anobject-matching score indicating a degree to which one or morecharacteristics (e.g., physical appearance-based characteristics,functionality characteristics, acoustic characteristics) of a physicalobject of a physical environment match one or more characteristics of avirtual object of an augmented reality experience. For example, inresponse to determining the characteristics of an identified physicalobject and identifying the virtual objects associated with an augmentedreality experience, the object-matching score generator 708 cancalculate an object-matching score for between the physical object andeach of the identified virtual objects.

In at least one embodiment, the object-matching score generator 708calculates the object-matching score associated with the physical objectand a particular virtual object by identifying matches (e.g., characterstring matches, threshold matches) between the characteristics of thephysical object and characteristics of the particular virtual object.For each identified match, the object-matching score generator 708 canadd a value or point to a total score for the object pair. Additionally,the object-matching score generator 708 can further weight the value orpoint based on the relevancy of the matched characteristics. Forexample, if the matched characteristics indicate an appearancesimilarity between the objects and/or a functionality similarity betweenthe objects, the object-matching score generator 708 can add a weight tothe value of point added to the total score for the object pair.

After calculating object-matching scores for every combination ofphysical objects in the physical environment and virtual objects in theaugmented reality experience, the object-matching score generator 708can identify analogous virtual objects. For example, for a particularphysical object in the physical environment, the object-matching scoregenerator 708 can identify the highest object-matching score associatedwith that physical object. The object-matching score generator 708 canfurther determine that the virtual object associated with that highscore is analogous to the physical object. In at least one embodiment,the object-matching score generator 708 can determine that the virtualobject is analogous to the physical object when the object-matchingscore associated with both is highest and when that score is satisfiesan object-matching threshold. The object-matching score generator 708can repeat this process for every physical object identified in thephysical environment.

As shown in FIG. 7 , and as mentioned above, the augmented realitysystem 102 also includes the anchor generator 710. In one or moreembodiments, the anchor generator 710 anchors one or more features of anaugmented reality experience to a physical object determined tocorrespond to an analogous virtual object of the augmented realityexperience. For example, the anchor generator 710 can identify visualand acoustic features of an augmented reality experience based on ananalysis of rendering and playback instructions associated with theaugmented reality experience.

To illustrate, if the augmented reality experience is one where the useris only meant to listen to music, the anchor generator 710 can identifyacoustic features of the music (e.g., the music file for playing, presetplayback levels, sound distortions and enhancements). The anchorgenerator 710 can further anchor those acoustic features to the physicalobject by associating those features with a location of the physicalobject, as indicated by the three-dimensional map of the physicalenvironment. The anchor generator 710 can repeat this process with othertypes of features associated with the augmented reality experience.

As shown in FIG. 7 , and as mentioned above, the augmented realitysystem 102 also includes the AR experience renderer 712. In one or moreembodiments, the AR experience renderer 712 generates an augmentedreality experience for display via the augmented-reality-computingdevice 106 a. For example, the AR experience renderer 712 can accessrendering instructions associated with the augmented reality experienceto render virtual objects including texture, lighting, and shadingaccording to the positioning of the virtual objects within the augmentedreality experience.

As further shown in FIG. 7 , and as mentioned above, the augmentedreality system 102 includes the feature modifier 714. In one or moreembodiments, the feature modifier 714 modifies one or more features ofan augmented reality experience based on those features being anchoredto a particular physical object within the physical environment. Forexample, the feature modifier 714 can modify acoustic features of asound of the augmented reality experience to simulate that the soundoriginates from the physical object. Additionally or alternatively, thefeature modifier 714 can modify the acoustic features of the sound tosimulate an effect on the sound by the physical object. Additionally oralternatively, the feature modifier 714 can modify or consolidate audiostreams of the sound based on the acoustic features being anchored tothe physical object. Additionally or alternatively, the feature modifier714 can modify the acoustic features of the sound based on a soundprofile of the analogous virtual object. As discussed above, the featuremodifier 714 can modify the acoustic features of the sound based on: adistance between a location of the physical object and theaugmented-reality-computing device 106 a, spectral localization cuesfrom the location of the physical object relative to theaugmented-reality-computing device 106 a, and/or a visual characteristicof the physical object.

As mentioned above, and as shown in FIG. 7 , the augmented realitysystem 102 includes the overlay generator 716. In one or moreembodiments, the overlay generator 716 identifies one or more visualcharacteristics of an analogous virtual object and generates a virtualgraphic overlay based on the identified visual characteristics. In atleast one embodiment, the overlay generator 716 further provides thegenerated virtual graphic overlay to the AR experience renderer 712 forinclusion in the augmented reality experience along with renderinginstructions to superimpose the virtual graphic overlay at a positionthat overlays a portion of the corresponding physical object or over theentire corresponding physical object. In additional or alternativeembodiments, the overlay generator 716 can generate updated or newvirtual graphic overlays based on detected user interactions.

As shown in FIG. 7 , and as mentioned above, the augmented realitysystem 102 includes an interaction tracker 718. In one or moreembodiments, the interaction tracker 718 detects and tracks userinteractions with virtual graphic overlays and physical objects. Forexample, the interaction tracker 718 can detect user interactions with avirtual graphic overlay, with an area of a physical object on which thevirtual graphic overlay is superimposed, and/or with the physical objectwith no virtual graphic overlay superimposed. Based on the detected userinteractions, the interaction tracker 718 can request additionalmodifications be performed by the feature modifier 714.

As further shown in FIG. 7 , the augmented reality system 102 includesthe data storage 720 including the object data 722, the physicalenvironment data 724, and the augmented reality experience data 726. Inone or more embodiments, the object data 722 includes informationassociated with physical objects and/or virtual objects such asdescribed herein (e.g., identifications, types, classifications,features, characteristics). In one or more embodiments, the physicalenvironment data 724 includes information associated with physicalenvironments such as described herein (e.g., 3D maps, localizations,relative distances, anchors). In one or more embodiments, the augmentedreality experience data 726 includes information associated withaugmented reality experiences such as described herein (e.g., requiredvirtual objects, associated augmented reality experience, positions,sounds).

Each of the components 702-726 of the augmented reality system 102 caninclude software, hardware, or both. For example, the components 702-726can include one or more instructions stored on a computer-readablestorage medium and executable by processors of one or more computingdevices, such as a client device or server device. When executed by theone or more processors, the computer-executable instructions of theaugmented reality system 102 can cause the computing device(s) toperform the methods described herein. Alternatively, the components702-726 can include hardware, such as a special-purpose processingdevice to perform a certain function or group of functions.Alternatively, the components 702-726 of the augmented reality system102 can include a combination of computer-executable instructions andhardware.

Furthermore, the components 702-726 of the augmented reality system 102may, for example, be implemented as one or more operating systems, asone or more stand-alone applications, as one or more modules of anapplication, as one or more plug-ins, as one or more library functionsor functions that may be called by other applications, and/or as acloud-computing model. Thus, the components 702-726 may be implementedas a stand-alone application, such as a desktop or mobile application.Furthermore, the components 702-726 may be implemented as one or moreweb-based applications hosted on a remote server. The components 702-726may also be implemented in a suite of mobile device applications or“apps.”

FIGS. 1-7 , the corresponding text, and the examples provide a number ofdifferent methods, systems, devices, and non-transitorycomputer-readable media of the augmented reality system 102. In additionto the foregoing, one or more embodiments can also be described in termsof flowcharts comprising acts for accomplishing a particular result, asshown in FIG. 8 . FIG. 8 may be performed with more or fewer acts.Further, the acts may be performed in differing orders. Additionally,the acts described herein may be repeated or performed in parallel withone another or parallel with different instances of the same or similaracts.

In accordance with one or more embodiments, FIG. 8 illustrates aflowchart of a series of acts 800 for determining a physical object froma physical environment corresponds to an analogous virtual object for anaugmented reality experience and modifying acoustic features of a soundfor the augmented reality experience to integrate the physical objectinto the augmented reality experience. While FIG. 8 illustrates actsaccording to one embodiment, alternative embodiments may omit, add to,reorder, and/or modify any of the acts shown in FIG. 8 . The acts ofFIG. 8 can be performed as part of a method. Alternatively, anon-transitory computer-readable medium can comprise instructions that,when executed by one or more processors, cause a computing device toperform the acts of FIG. 8 . In some embodiments, a system can performthe acts of FIG. 8 .

As shown in FIG. 8 , the series of acts 800 includes an act 810 ofcapturing a data stream corresponding to a physical environment. Forexample, the act 810 can involve capturing a data stream correspondingto a physical environment utilizing an augmented-reality-computingdevice. As further shown in FIG. 8 , the series of acts 800 includes anact 820 of determining that a physical object within the physicalenvironment corresponds to an analogous virtual object of an augmentedreality experience. For example, determining that the physical objectwithin the physical environment corresponds to the analogous virtualobject of an augmented reality experience can be based on imagecomparisons, description comparisons, heat maps, and/or machinelearning. In one or more embodiments, determining that the physicalobject within the physical environment corresponds to the analogousvirtual object of the augmented reality experience includes: generatingan object-matching score indicating a degree to which one or morecharacteristics of the physical object match one or more characteristicsof the analogous virtual object; and determining the object-matchingscore satisfies an object-matching threshold.

As shown in FIG. 8 , the series of acts 800 includes an act 830 ofmodifying one or more acoustic features of a sound for the augmentedreality experience to simulate that the sound originates from thephysical object or to simulate an effect on the sound. For example, theact 830 can involve modifying, by the augmented-reality-computingdevice, one or more acoustic features of a sound for the augmentedreality experience to simulate that the sound originates from thephysical object or to simulate an effect on the sound by the physicalobject. In at least one embodiment, the series of acts 800 furtherincludes mapping the physical environment to determine a location of thephysical object relative to the augmented-reality-computing device. Forexample, modifying the one or more acoustic features of the sound caninclude modifying the sound to simulate the sound originating from thelocation of the physical object relative to theaugmented-reality-computing device.

In one or more embodiments, modifying the one or more acoustic featuresof the sound can include one or more of: modifying an acoustic featureof the sound based on a distance between a location of the physicalobject and the augmented-reality-computing device; modifying theacoustic feature of the sound based on spectral localization cues fromthe location of the physical object relative to theaugmented-reality-computing device; or modifying the acoustic feature ofthe sound based on a visual characteristic of the physical object. In atleast one embodiment, modifying the one or more acoustic features of thesound includes one or more of: modifying one or more audio streamscorresponding to the sound for the augmented reality experience; orconsolidating two or more audio streams corresponding to the sound forthe augmented reality experience. For example, modifying the one or moreacoustic features of the sound can include: identifying a sound profileassociated with the analogous virtual object; and modifying an acousticfeature of the sound based on the sound profile associated with theanalogous virtual object.

As shown in FIG. 8 , the series of acts 800 includes an act 840 ofpresenting the augmented reality experience without utilizing theanalogous virtual object. For example, the act 840 can involvepresenting, by the augmented-reality-computing device, the augmentedreality experience without utilizing the analogous virtual object. Forexample, presenting the augmented reality experience without utilizingthe analogous virtual object can include rendering the augmented realityexperience utilizing the physical object instead of the analogousvirtual object.

In one or more embodiments, the series of acts 800 includes acts of:identifying a visual characteristic of the analogous virtual object;generating a virtual graphic overlay based on the visual characteristic;and presenting the augmented reality experience by superimposing thevirtual graphic overlay over a portion of the physical object or over anentirety of the physical object. In at least one embodiment, the seriesof acts 800 includes: detecting a user interaction with an area of thephysical object on which the virtual graphic overlay is superimposed;generating a new virtual graphic overlay based on the user interaction;and rendering the new virtual graphic overlay superimposed over theportion of the physical object or over the entirety of the physicalobject.

Additionally, in one or more embodiments, the series of acts 800includes: identifying that the sound corresponds to an additionalvirtual object from the augmented reality experience; identifying asound effect for the sound based on the analogous virtual object;determining a physical characteristic of the physical object; andmodifying the one or more acoustic features of the sound to simulate thesound effect based on the physical characteristic of the physicalobject. For example, the series of acts 800 can further includedetermining the physical characteristic of the physical object bydetermining one or more of: a thickness of the physical object, a massof the physical object, a size of the physical object, a shape of thephysical object, or a density of the physical object. The series of acts800 can also include determining that the physical object displays oneor more images or produces audio.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory), and executes those instructions, thereby performingone or more processes, including one or more of the processes describedherein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general-purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed on ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. In this description, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources. For example, cloud computingcan be employed in the marketplace to offer ubiquitous and convenienton-demand access to the shared pool of configurable computing resources.The shared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the claims, a “cloud-computingenvironment” is an environment in which cloud computing is employed.

FIG. 9 illustrates a block diagram of exemplary computing device 900that may be configured to perform one or more of the processes describedabove. One will appreciate that one or more computing devices such asthe computing device 900 may implement the augmented reality system 102.As shown by FIG. 9 , the computing device 900 can comprise a processor902, a memory 904, a storage device 906, an I/O interface 908, and acommunication interface 910, which may be communicatively coupled by wayof a communication infrastructure 912. While an exemplary computingdevice 900 is shown in FIG. 9 , the components illustrated in FIG. 9 arenot intended to be limiting. Additional or alternative components may beused in other embodiments. Furthermore, in certain embodiments, thecomputing device 900 can include fewer components than those shown inFIG. 9 . Components of the computing device 900 shown in FIG. 9 will nowbe described in additional detail.

In one or more embodiments, the processor 902 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions, theprocessor 902 may retrieve (or fetch) the instructions from an internalregister, an internal cache, the memory 904, or the storage device 906and decode and execute them. In one or more embodiments, the processor902 may include one or more internal caches for data, instructions, oraddresses. As an example and not by way of limitation, the processor 902may include one or more instruction caches, one or more data caches, andone or more translation lookaside buffers (TLBs). Instructions in theinstruction caches may be copies of instructions in the memory 904 orthe storage device 906.

The memory 904 may be used for storing data, metadata, and programs forexecution by the processor(s). The memory 904 may include one or more ofvolatile and non-volatile memories, such as Random Access Memory(“RAM”), Read Only Memory (“ROM”), a solid-state disk (“SSD”), Flash,Phase Change Memory (“PCM”), or other types of data storage. The memory904 may be internal or distributed memory.

The storage device 906 includes storage for storing data orinstructions. As an example and not by way of limitation, storage device906 can comprise a non-transitory storage medium described above. Thestorage device 906 may include a hard disk drive (HDD), a floppy diskdrive, flash memory, an optical disc, a magneto-optical disc, magnetictape, or a Universal Serial Bus (USB) drive or a combination of two ormore of these. The storage device 906 may include removable ornon-removable (or fixed) media, where appropriate. The storage device906 may be internal or external to the computing device 900. In one ormore embodiments, the storage device 906 is non-volatile, solid-statememory. In other embodiments, the storage device 906 includes read-onlymemory (ROM). Where appropriate, this ROM may be mask programmed ROM,programmable ROM (PROM), erasable PROM (EPROM), electrically erasablePROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or acombination of two or more of these.

The I/O interface 908 allows a user to provide input to, receive outputfrom, and otherwise transfer data to and receive data from computingdevice 900. The I/O interface 908 may include a mouse, a keypad or akeyboard, a touch screen, a camera, an optical scanner, networkinterface, modem, other known I/O devices or a combination of such I/Ointerfaces. The I/O interface 908 may include one or more devices forpresenting output to a user, including, but not limited to, a graphicsengine, a display (e.g., a display screen), one or more output drivers(e.g., display drivers), one or more audio speakers, and one or moreaudio drivers. In certain embodiments, the I/O interface 908 isconfigured to provide graphical data to a display for presentation to auser. The graphical data may be representative of one or more graphicaluser interfaces and/or any other graphical content as may serve aparticular implementation.

The communication interface 910 can include hardware, software, or both.In any event, the communication interface 910 can provide one or moreinterfaces for communication (such as, for example, packet-basedcommunication) between the computing device 900 and one or more othercomputing devices or networks. As an example and not by way oflimitation, the communication interface 910 may include a networkinterface controller (NIC) or network adapter for communicating with anEthernet or other wire-based network or a wireless NIC (WNIC) orwireless adapter for communicating with a wireless network, such as aWI-FI.

Additionally or alternatively, the communication interface 910 mayfacilitate communications with an ad hoc network, a personal areanetwork (PAN), a local area network (LAN), a wide area network (WAN), ametropolitan area network (MAN), or one or more portions of the Internetor a combination of two or more of these. One or more portions of one ormore of these networks may be wired or wireless. As an example, thecommunication interface 910 may facilitate communications with awireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FInetwork, a WI-MAX network, a cellular telephone network (such as, forexample, a Global System for Mobile Communications (GSM) network), orother suitable wireless network or a combination thereof.

Additionally, the communication interface 910 may facilitatecommunications various communication protocols. Examples ofcommunication protocols that may be used include, but are not limitedto, data transmission media, communications devices, TransmissionControl Protocol (“TCP”), Internet Protocol (“IP”), File TransferProtocol (“FTP”), Telnet, Hypertext Transfer Protocol (“HTTP”),Hypertext Transfer Protocol Secure (“HTTPS”), Session InitiationProtocol (“SIP”), Simple Object Access Protocol (“SOAP”), ExtensibleMark-up Language (“XML”) and variations thereof, Simple Mail TransferProtocol (“SMTP”), Real-Time Transport Protocol (“RTP”), User DatagramProtocol (“UDP”), Global System for Mobile Communications (“GSM”)technologies, Code Division Multiple Access (“CDMA”) technologies, TimeDivision Multiple Access (“TDMA”) technologies, Short Message Service(“SMS”), Multimedia Message Service (“MMS”), radio frequency (“RF”)signaling technologies, Long Term Evolution (“LTE”) technologies,wireless communication technologies, in-band and out-of-band signalingtechnologies, and other suitable communications networks andtechnologies.

The communication infrastructure 912 may include hardware, software, orboth that couples components of the computing device 900 to each other.As an example and not by way of limitation, the communicationinfrastructure 912 may include an Accelerated Graphics Port (AGP) orother graphics bus, an Enhanced Industry Standard Architecture (EISA)bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, anIndustry Standard Architecture (ISA) bus, an INFINIBAND interconnect, alow-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture(MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express(PCIe) bus, a serial advanced technology attachment (SATA) bus, a VideoElectronics Standards Association local (VLB) bus, or another suitablebus or a combination thereof.

As mentioned above, the augmented reality system 102 can be implementedas part of (or including) a networking system. In one or moreembodiments, the networking system comprises a social networking system.In addition to the description given above, a social networking systemmay enable its users (such as persons or organizations) to interact withthe system and with each other. The social networking system may, withinput from a user, create and store in the social networking system auser profile associated with the user. The user profile may includedemographic information, communication-channel information, andinformation on personal interests of the user. The social networkingsystem may also, with input from a user, create and store a record ofrelationships of the user with other users of the social networkingsystem, as well as provide services (e.g., posts, photo-sharing, eventorganization, messaging, games, or advertisements) to facilitate socialinteraction between or among users.

The social networking system may store records of users andrelationships between users in a social graph comprising a plurality ofnodes and a plurality of edges connecting the nodes. The nodes maycomprise a plurality of user nodes and a plurality of concept nodes. Auser node of the social graph may correspond to a user of the socialnetworking system. A user may be an individual (human user), an entity(e.g., an enterprise, business, or third party application), or a group(e.g., of individuals or entities). A user node corresponding to a usermay comprise information provided by the user and information gatheredby various systems, including the social networking system.

For example, the user may provide his or her name, profile picture, cityof residence, contact information, birth date, gender, marital status,family status, employment, educational background, preferences,interests, and other demographic information to be included in the usernode. Each user node of the social graph may have a corresponding webpage (typically known as a profile page). In response to a requestincluding a user name, the social networking system can access a usernode corresponding to the user name, and construct a profile pageincluding the name, a profile picture, and other information associatedwith the user. A profile page of a first user may display to a seconduser all or a portion of the first user's information based on one ormore privacy settings by the first user and the relationship between thefirst user and the second user.

A concept node may correspond to a concept of the social networkingsystem. For example, a concept can represent a real-world entity, suchas a movie, a song, a sports team, a celebrity, a group, a restaurant,or a place or a location. An administrative user of a concept nodecorresponding to a concept may create or update the concept node byproviding information of the concept (e.g., by filling out an onlineform), causing the social networking system to associate the informationwith the concept node. For example and without limitation, informationassociated with a concept can include a name or a title, one or moreimages (e.g., an image of cover page of a book), a web site (e.g., anURL address) or contact information (e.g., a phone number, an emailaddress). Each concept node of the social graph may correspond to a webpage. For example, in response to a request including a name, the socialnetworking system can access a concept node corresponding to the name,and construct a web page including the name and other informationassociated with the concept.

An edge between a pair of nodes may represent a relationship between thepair of nodes. For example, an edge between two user nodes can representa friendship between two users. For another example, the socialnetworking system may construct a web page (or a structured document) ofa concept node (e.g., a restaurant, a celebrity), incorporating one ormore selectable option or selectable elements (e.g., “like”, “check in”)in the web page. A user can access the page using a web browser hostedby the user's client device and select a selectable option or selectableelement, causing the client device to transmit to the social networkingsystem a request to create an edge between a user node of the user and aconcept node of the concept, indicating a relationship between the userand the concept (e.g., the user checks in a restaurant, or the user“likes” a celebrity).

As an example, a user may provide (or change) his or her city ofresidence, causing the social networking system to create an edgebetween a user node corresponding to the user and a concept nodecorresponding to the city declared by the user as his or her city ofresidence. In addition, the degree of separation between any two nodesis defined as the minimum number of hops required to traverse the socialgraph from one node to the other. A degree of separation between twonodes can be considered a measure of relatedness between the users orthe concepts represented by the two nodes in the social graph. Forexample, two users having user nodes that are directly connected by anedge (i.e., are first-degree nodes) may be described as “connectedusers” or “friends.” Similarly, two users having user nodes that areconnected only through another user node (i.e., are second-degree nodes)may be described as “friends of friends.”

A social networking system may support a variety of applications, suchas photo sharing, on-line calendars and events, gaming, instantmessaging, and advertising. For example, the social networking systemmay also include media sharing capabilities. Also, the social networkingsystem may allow users to post photographs and other multimedia contentitems to a user's profile page (typically known as “wall posts” or“timeline posts”) or in a photo album, both of which may be accessibleto other users of the social networking system depending upon the user'sconfigured privacy settings. The social networking system may also allowusers to configure events. For example, a first user may configure anevent with attributes including time and date of the event, location ofthe event and other users invited to the event. The invited users mayreceive invitations to the event and respond (such as by accepting theinvitation or declining it). Furthermore, the social networking systemmay allow users to maintain a personal calendar. Similarly to events,the calendar entries may include times, dates, locations and identitiesof other users.

FIG. 10 illustrates an example network environment 1000 of an augmentedreality system. Network environment 1000 includes a client system 1008,an augmented reality system 1002 (e.g., the augmented reality system102), and a third-party system 1006 connected to each other by a network1004. Although FIG. 10 illustrates a particular arrangement of theclient system 1008, augmented reality system 1002, third-party system1006, and network 1004, this disclosure contemplates any suitablearrangement of the client system 1008, augmented reality system 1002,third-party system 1006, and network 1004. As an example and not by wayof limitation, two or more of client system 1008, augmented realitysystem 1002, and third-party system 1006 may be connected to each otherdirectly, bypassing network 1004. As another example, two or more of theclient system 1008, augmented reality system 1002, and third-partysystem 1006 may be physically or logically co-located with each other inwhole or in part. Moreover, although FIG. 10 illustrates a particularnumber of client systems 1008, networking systems 1002, third-partysystems 1006, and networks 1004, this disclosure contemplates anysuitable number of client systems 1008, augmented reality system 1002,third-party systems 1006, and networks 1004. As an example and not byway of limitation, network environment 1000 may include multiple clientsystems 1008, augmented reality systems 1002, third-party systems 1006,and networks 1004.

This disclosure contemplates any suitable network 1004. As an exampleand not by way of limitation, one or more portions of network 1004 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 1004 may include one or more networks1004.

Links may connect the client system 1008, augmented reality system 1002,and third-party system 1006 to communication network 1004 or to eachother. This disclosure contemplates any suitable links. In particularembodiments, one or more links include one or more wireline (such as forexample Digital Subscriber Line (DSL) or Data Over Cable ServiceInterface Specification (DOCSIS)), wireless (such as for example Wi-Fior Worldwide Interoperability for Microwave Access (WiMAX)), or optical(such as for example Synchronous Optical Network (SONET) or SynchronousDigital Hierarchy (SDH)) links. In particular embodiments, one or morelinks each include an ad hoc network, an intranet, an extranet, a VPN, aLAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portionof the PSTN, a cellular technology-based network, a satellitecommunications technology-based network, another link, or a combinationof two or more such links. Links need not necessarily be the samethroughout network environment 1000. One or more first links may differin one or more respects from one or more second links.

In particular embodiments, the client system 1008 may be an electronicdevice including hardware, software, or embedded logic components or acombination of two or more such components and capable of carrying outthe appropriate functionalities implemented or supported by the clientsystem 1008. As an example and not by way of limitation, a client system1008 may include a computer system such as an augmented reality displaydevice, a desktop computer, notebook or laptop computer, netbook, atablet computer, e-book reader, GPS device, camera, personal digitalassistant (PDA), handheld electronic device, cellular telephone,smartphone, other suitable electronic device, or any suitablecombination thereof. This disclosure contemplates any suitable clientsystems 1008. A client system 1008 may enable a network user at theclient system 1008 to access network 1004. A client system 1008 mayenable its user to communicate with other users at other client devices1008.

In particular embodiments, the client system 1008 may include a webbrowser, such as MICROSOFT INTERNET EXPLORER, GOOGLE CHROME or MOZILLAFIREFOX, and may have one or more add-ons, plug-ins, or otherextensions, such as TOOLBAR or YAHOO TOOLBAR. A user at the clientsystem 1008 may enter a Uniform Resource Locator (URL) or other addressdirecting the web browser to a particular server (such as server, or aserver associated with a third-party system 1006), and the web browsermay generate a Hyper Text Transfer Protocol (HTTP) request andcommunicate the HTTP request to server. The server may accept the HTTPrequest and communicate to the client system 1008 one or more Hyper TextMarkup Language (HTML) files responsive to the HTTP request. The clientsystem 1008 may render a webpage based on the HTML files from the serverfor presentation to the user. This disclosure contemplates any suitablewebpage files. As an example, and not by way of limitation, webpages mayrender from HTML files, Extensible Hyper Text Markup Language (XHTML)files, or Extensible Markup Language (XML) files, according toparticular needs. Such pages may also execute scripts such as, forexample and without limitation, those written in JAVASCRIPT, JAVA,MICROSOFT SILVERLIGHT, combinations of markup language and scripts suchas AJAX (Asynchronous JAVASCRIPT and XML), and the like. Herein,reference to a webpage encompasses one or more corresponding webpagefiles (which a browser may use to render the webpage) and vice versa,where appropriate.

In particular embodiments, augmented reality system 1002 may be anetwork-addressable computing system that can host an online augmentedreality system. Augmented reality system 1002 may generate, store,receive, and send augmented reality data, such as, for example,augmented reality scenes, augmented reality experiences, virtualobjects, or other suitable data related to the augmented reality system1002. Augmented reality system 1002 may be accessed by the othercomponents of network environment 1000 either directly or via network1004. In particular embodiments, augmented reality system 1002 mayinclude one or more servers. Each server may be a unitary server or adistributed server spanning multiple computers or multiple datacenters.Servers may be of various types, such as, for example and withoutlimitation, web server, news server, mail server, message server,advertising server, file server, application server, exchange server,database server, proxy server, another server suitable for performingfunctions or processes described herein, or any combination thereof. Inparticular embodiments, each server may include hardware, software, orembedded logic components or a combination of two or more suchcomponents for carrying out the appropriate functionalities implementedor supported by server. In particular embodiments, augmented realitysystem 1002 may include one or more data stores. Data stores may be usedto store various types of information. In particular embodiments, theinformation stored in data stores may be organized according to specificdata structures. In particular embodiments, each data store may be arelational, columnar, correlation, or other suitable database. Althoughthis disclosure describes or illustrates particular types of databases,this disclosure contemplates any suitable types of databases. Particularembodiments may provide interfaces that enable a client system 1008, anaugmented reality system 1002, or a third-party system 1006 to manage,retrieve, modify, add, or delete, the information stored in data store.

In particular embodiments, augmented reality system 1002 may store oneor more social graphs in one or more data stores. In particularembodiments, a social graph may include multiple nodes—which may includemultiple user nodes (each corresponding to a particular user) ormultiple concept nodes (each corresponding to a particular concept)—andmultiple edges connecting the nodes. Augmented reality system 1002 mayprovide users of the online social network the ability to communicateand interact with other users. In particular embodiments, users may jointhe online social network via augmented reality system 1002 and then addconnections (e.g., relationships) to a number of other users ofaugmented reality system 1002 that they want to be connected to. Herein,the term “friend” may refer to any other user of augmented realitysystem 1002 with whom a user has formed a connection, association, orrelationship via augmented reality system 1002.

In particular embodiments, augmented reality system 1002 may provideusers with the ability to take actions on various types of items orobjects, supported by augmented reality system 1002. As an example andnot by way of limitation, the items and objects may include groups orsocial networks to which users of augmented reality system 1002 maybelong, events or calendar entries in which a user might be interested,computer-based applications that a user may use, transactions that allowusers to buy or sell items via the service, interactions withadvertisements that a user may perform, or other suitable items orobjects. A user may interact with anything that is capable of beingrepresented in augmented reality system 1002 or by an external system ofthird-party system 1006, which is separate from augmented reality system1002 and coupled to augmented reality system 1002 via a network 1004.

In particular embodiments, augmented reality system 1002 may be capableof linking a variety of entities. As an example and not by way oflimitation, augmented reality system 1002 may enable users to interactwith each other as well as receive content from third-party systems 1006or other entities, or to allow users to interact with these entitiesthrough an application programming interfaces (API) or othercommunication channels.

In particular embodiments, a third-party system 1006 may include one ormore types of servers, one or more data stores, one or more interfaces,including but not limited to APIs, one or more web services, one or morecontent sources, one or more networks, or any other suitable components,e.g., that servers may communicate with. A third-party system 1006 maybe operated by a different entity from an entity operating augmentedreality system 1002. In particular embodiments, however, augmentedreality system 1002 and third-party systems 1006 may operate inconjunction with each other to provide social-networking services tousers of augmented reality system 1002 or third-party systems 1006. Inthis sense, augmented reality system 1002 may provide a platform, orbackbone, which other systems, such as third-party systems 1006, may useto provide social-networking services and functionality to users acrossthe Internet.

In particular embodiments, a third-party system 1006 may include athird-party content object provider. A third-party content objectprovider may include one or more sources of content objects, which maybe communicated to a client system 1008. As an example, and not by wayof limitation, content objects may include information regarding thingsor activities of interest to the user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, productinformation and reviews, or other suitable information. As anotherexample and not by way of limitation, content objects may includeincentive content objects, such as coupons, discount tickets, giftcertificates, or other suitable incentive objects.

In particular embodiments, augmented reality system 1002 also includesuser-generated content objects, which may enhance a user's interactionswith augmented reality system 1002. User-generated content may includeanything a user can add, upload, send, or “post” to augmented realitysystem 1002. As an example and not by way of limitation, a usercommunicates posts to augmented reality system 1002 from a client system1008. Posts may include data such as status updates or other textualdata, location information, photos, videos, links, music or othersimilar data or media. Content may also be added to augmented realitysystem 1002 by a third-party through a “communication channel,” such asa newsfeed or stream.

In particular embodiments, augmented reality system 1002 may include avariety of servers, sub-systems, programs, modules, logs, and datastores. In particular embodiments, augmented reality system 1002 mayinclude one or more of the following: a web server, action logger,API-request server, relevance-and-ranking engine, content-objectclassifier, notification controller, action log,third-party-content-object-exposure log, inference module,authorization/privacy server, search module, advertisement-targetingmodule, user-interface module, user-profile store, connection store,third-party content store, or location store. Augmented reality system1002 may also include suitable components such as network interfaces,security mechanisms, load balancers, failover servers,management-and-network-operations consoles, other suitable components,or any suitable combination thereof. In particular embodiments,augmented reality system 1002 may include one or more user-profilestores for storing user profiles. A user profile may include, forexample, biographic information, demographic information, behavioralinformation, social information, or other types of descriptiveinformation, such as work experience, educational history, hobbies orpreferences, interests, affinities, or location. Interest informationmay include interests related to one or more categories. Categories maybe general or specific. As an example and not by way of limitation, if auser “likes” an article about a brand of shoes the category may be thebrand, or the general category of “shoes” or “clothing.” A connectionstore may be used for storing connection information about users. Theconnection information may indicate users who have similar or commonwork experience, group memberships, hobbies, educational history, or arein any way related or share common attributes. The connectioninformation may also include user-defined connections between differentusers and content (both internal and external). A web server may be usedfor linking augmented reality system 1002 to one or more client system1008 or one or more third-party system 1006 via network 1004. The webserver may include a mail server or other messaging functionality forreceiving and routing messages between augmented reality system 1002 andone or more client systems 1008. An API-request server may allow athird-party system 1006 to access information from augmented realitysystem 1002 by calling one or more APIs. An action logger may be used toreceive communications from a web server about a user's actions on oroff augmented reality system 1002. In conjunction with the action log, athird-party-content-object log may be maintained of user exposures tothird-party-content objects. A notification controller may provideinformation regarding content objects to a client system 1008.Information may be pushed to a client system 1008 as notifications, orinformation may be pulled from client system 1008 responsive to arequest received from client system 1008. Authorization servers may beused to enforce one or more privacy settings of the users of augmentedreality system 1002. A privacy setting of a user determines howparticular information associated with a user can be shared. Theauthorization server may allow users to opt in to or opt out of havingtheir actions logged by augmented reality system 1002 or shared withother systems (e.g., third-party system 1006), such as, for example, bysetting appropriate privacy settings. Third-party-content-object storesmay be used to store content objects received from third parties, suchas a third-party system 1006. Location stores may be used for storinglocation information received from client system 1008 associated withusers. Advertisement-pricing modules may combine social information, thecurrent time, location information, or other suitable information toprovide relevant advertisements, in the form of notifications, to auser.

The foregoing specification is described with reference to specificexemplary embodiments thereof. Various embodiments and aspects of thedisclosure are described with reference to details discussed herein, andthe accompanying drawings illustrate the various embodiments. Thedescription above and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding of various embodiments.

The additional or alternative embodiments may be embodied in otherspecific forms without departing from its spirit or essentialcharacteristics. The described embodiments are to be considered in allrespects only as illustrative and not restrictive. The scope of theinvention is, therefore, indicated by the appended claims rather than bythe foregoing description. All changes that come within the meaning andrange of equivalency of the claims are to be embraced within theirscope.

What is claimed is:
 1. A computer-implemented method comprising: capturing a data stream corresponding to a physical environment utilizing an augmented-reality-computing device; identifying a physical object within the physical environment that corresponds to an analogous virtual object of an augmented reality experience; analyzing the analogous virtual object within the augmented reality experience to determine a function of one or more acoustic features of the analogous virtual object; modifying, by the augmented-reality-computing device, one or more acoustic features of a sound for the augmented reality experience based on the function of one or more acoustic features of the analogous virtual object to simulate that the sound for the augmented reality experience originates from a location of the physical object; modifying, by the augmented-reality-computing device, one or more visual features of the augmented reality experience by applying a visual overlay to the location of the physical object; and presenting, by the augmented-reality-computing device, the augmented reality experience with the modified one or more acoustic features and the analogous visual object.
 2. The computer-implemented method as recited in claim 1, further comprising: mapping the physical environment to determine a location of the physical object relative to the augmented-reality-computing device, wherein the augmented-reality-computing device comprises a mixed reality headset; and wherein modifying the one or more acoustic features of the sound for the augmented reality experience comprises modifying the sound for the augmented reality experience to simulate the sound originating from the location of the physical object relative to the augmented-reality-computing device.
 3. The computer-implemented method as recited in claim 1, further comprising modifying the one or more acoustic features of the sound for the augmented reality experience by performing one or more of: modifying an acoustic feature of the sound for the augmented reality experience based on a distance between the location of the physical object and the augmented-reality-computing device; modifying the acoustic feature of the sound for the augmented reality experience based on spectral localization cues from the location of the physical object relative to the augmented-reality-computing device; or modifying the acoustic feature of the sound for the augmented reality experience based on a visual characteristic of the physical object.
 4. The computer-implemented method as recited in claim 1, further comprising modifying the one or more acoustic features of the sound for the augmented reality experience by performing one or more of: modifying one or more audio streams corresponding to the sound for the augmented reality experience; or consolidating two or more audio streams corresponding to the sound for the augmented reality experience.
 5. The computer-implemented method as recited in claim 4, further comprising modifying the one or more acoustic features of the sound for the augmented reality experience by: identifying a sound profile associated with the analogous virtual object; and modifying an acoustic feature of the sound for the augmented reality experience based on the sound profile associated with the analogous virtual object.
 6. The computer-implemented method as recited in claim 1, further comprising: identifying a visual characteristic of the analogous virtual object; generating a virtual graphic overlay based on the visual characteristic; and presenting the augmented reality experience by superimposing the virtual graphic overlay over a portion of the physical object or over an entirety of the physical object.
 7. The computer-implemented method as recited in claim 6, further comprising: detecting a user interaction with the location of the physical object on which the virtual graphic overlay is superimposed; generating a new virtual graphic overlay based on the user interaction with the location of the physical object; and rendering the new virtual graphic overlay superimposed over the portion of the physical object or over the entirety of the physical object.
 8. The computer-implemented method as recited in claim 1, further comprising: identifying that the sound for the augmented reality experience corresponds to an additional virtual object from the augmented reality experience; identifying a sound effect for the sound based on the analogous virtual object; determining a physical characteristic of the physical object; and modifying the one or more acoustic features of the sound to simulate the sound effect based on the physical characteristic of the physical object.
 9. The computer-implemented method as recited in claim 8, wherein determining the physical characteristic of the physical object comprises determining one or more of: a thickness of the physical object, a mass of the physical object, a size of the physical object, a shape of the physical object, or a density of the physical object.
 10. The computer-implemented method as recited in claim 1, wherein determining that the physical object within the physical environment corresponds to the analogous virtual object of the augmented reality experience comprises: generating an object-matching score indicating a degree to which one or more characteristics of the physical object match one or more characteristics of the analogous virtual object; and determining the object-matching score satisfies an object-matching threshold.
 11. A system comprising: at least one processor; and at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to: capture a data stream corresponding to a physical environment utilizing an augmented-reality-computing device; identify that a physical object within the physical environment that corresponds to an analogous virtual object of an augmented reality experience; analyze the analogous virtual object within the augmented reality experience to determine a function of one or more acoustic features of the analogous virtual object; modifying, by the augmented-reality-computing device, one or more acoustic features of a sound for the augmented reality experience based on the function of one or more acoustic features of the analogous virtual object to simulate that the sound for the augmented reality experience originates from a location of the physical object; modifying, by the augmented-reality-computing device, one or more visual features of the augmented reality experience by applying a visual overlay to the location of the physical object; and presenting, by the augmented-reality-computing device, the augmented reality experience with the modified one or more acoustic features and the analogous visual object.
 12. The system as recited in claim 11, further storing instructions thereon that, when executed by the at least one processor, cause the system to: map the physical environment to determine a location of the physical object relative to the augmented-reality-computing device wherein the augmented-reality-computing device comprises a mixed reality headset; and modify the one or more acoustic features of the sound for the augmented reality experience by modifying the sound to simulate the sound originating from the location of the physical object relative to the augmented-reality-computing device.
 13. The system as recited in claim 11, further storing instructions thereon that, when executed by the at least one processor, cause the system to modify the one or more acoustic features of the sound for the augmented reality experience by performing one or more of: modifying an acoustic feature of the sound for the augmented reality experience based on a distance between the location of the physical object and the augmented-reality-computing device; modifying the acoustic feature of the sound for the augmented reality experience based on spectral localization cues from the location of the physical object relative to the augmented-reality-computing device; or modifying the acoustic feature of the sound for the augmented reality experience based on a visual characteristic of the physical object.
 14. The system as recited in claim 11, further storing instructions thereon that, when executed by the at least one processor, cause the system to modify the one or more acoustic features of the sound for the augmented reality experience by performing one or more of: modifying one or more audio streams corresponding to the sound for the augmented reality experience; or consolidating two or more audio streams corresponding to the sound for the augmented reality experience.
 15. The system as recited in claim 11, further storing instructions thereon that, when executed by the at least one processor, cause the system to modify the one or more acoustic features of the sound for the augmented reality experience by: identifying a sound profile associated with the analogous virtual object; and modifying an acoustic feature of the sound based on the sound profile associated with the analogous virtual object.
 16. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to: capture a data stream corresponding to a physical environment utilizing an augmented-reality-computing device; identify that a physical object within the physical environment that corresponds to an analogous virtual object of an augmented reality experience; analyze the analogous virtual object within the augmented reality experience to determine a function of one or more acoustic features of the analogous virtual object; modifying, by the augmented-reality-computing device, one or more acoustic features of a sound for the augmented reality experience based on the function of one or more acoustic features of the analogous virtual object to simulate that the sound for the augmented reality experience originates from a location of the physical object; modifying, by the augmented-reality-computing device, one or more visual features of the augmented reality experience by applying a visual overlay to the location of the physical object; and presenting, by the augmented-reality-computing device, the augmented reality experience with the modified one or more acoustic features and the analogous visual object.
 17. The non-transitory computer-readable medium as recited in claim 16, further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to: identify a visual characteristic of the analogous virtual object; generate a virtual graphic overlay based on the visual characteristic; and present the augmented reality experience by superimposing the virtual graphic overlay over a portion of the physical object or over an entirety of the physical object.
 18. The non-transitory computer-readable medium as recited in claim 17, further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to: detect a user interaction with an area of the physical object on which the virtual graphic overlay is superimposed; generate a new virtual graphic overlay based on the user interaction; and render the new virtual graphic overlay superimposed over the portion of the physical object or over the entirety of the physical object.
 19. The non-transitory computer-readable medium as recited in claim 16, further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to: identify that the sound for the augmented reality experience corresponds to an additional virtual object from the augmented reality experience, wherein the augmented-reality-computing device comprises a mixed reality headset; identify a sound effect for the sound based on the analogous virtual object; determine a physical characteristic of the physical object; and modify the one or more acoustic features of the sound for the augmented reality experience to simulate the sound effect based on the physical characteristic of the physical object.
 20. The non-transitory computer-readable medium as recited in claim 19, further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to further determine the physical characteristic of the physical object by determining that the physical object displays one or more images or produces audio. 